検索

elastiflowを使ってみる

Posted on | By ちぇけ

自宅でnetflowとかsflowでネットワークのトラフィックを監視したかったのでdockerで構築してみた。netflow analyzerとかntopの無料版を使ってみたけどダッシュボードはelastiflowのほうが圧倒的にダッシュボードがかっこよかった。本記事は備忘録レベルでわいだけ分かればおｋ的な感じで超雑にオナニーの合間に書いてるのでドキュメントとしてあてにしないように。

elastiflowはkibana + elasticsearch + elastiflowの3つで構成されてるOSSのフローコレクターらしい…。githubと公式ページ見た感じだと、elastiflowの部分は昔はlogstashだったっぽいけど今は専用のコレクターに変わってるのかな？筆者はELK Stackのことなんも知らないのでよくわからん。

とりあえずdocker-composeを作ってみた

githubの方は最新の構成じゃねぇとのことなので公式ページ参考に以下のように作ってみた。upして、ブラウザでkibananのuiにアクセスしても”Kibana server is not ready yet”って表示されてうまくいかなかった。elasticsearchのログを見てみると”Unable to retrieve version information from Elasticsearch nodes”というログを吐いてた。調べてみてもよくわからんかったので、teratailで質問したけど解答こなかった。

version: '3'
services:
  kibana:
    image: docker.elastic.co/kibana/kibana:7.13.1
    restart: unless-stopped
    hostname: kibana
    # network_mode: bridge
    networks:
      elastic:
      proxy_network:
    ports:
      # HTTP/REST
      - 5601:5601/tcp
    environment:
      TZ: Asia/Tokyo
      VIRTUAL_HOST: hogehoge.com
      VIRTUAL_PORT: 5601
      
      TELEMETRY_ENABLED: 'false'
      NEWSFEED_ENABLED: 'false'
      SERVER_NAME: 'kibana'
      SERVER_HOST: '0.0.0.0'
      SERVER_PORT: 5601
      SERVER_MAXPAYLOADBYTES: 8388608
      ELASTICSEARCH_HOSTS: 'http://elasticsearch:9200'
      ELASTICSEARCH_USERNAME: 'kibana_system'
      ELASTICSEARCH_PASSWORD: 'CHANGEME'
      ELASTICSEARCH_REQUESTTIMEOUT: 132000
      ELASTICSEARCH_SHARDTIMEOUT: 120000
      ELASTICSEARCH_SSL_VERIFICATIONMODE: 'none' 
      KIBANA_AUTOCOMPLETETIMEOUT: 3000
      KIBANA_AUTOCOMPLETETERMINATEAFTER: 2500000
      VIS_TYPE_VEGA_ENABLEEXTERNALURLS: 'true'
      XPACK_MAPS_SHOWMAPVISUALIZATIONTYPES: 'true'
      XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: 'ElastiFlow_0123456789_0123456789_0123456789'
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.1
    container_name: elasticsearch
    restart: unless-stopped
    hostname: elasticsearch
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 131072
        hard: 131072
      nproc: 8192
      fsize: -1
    networks:
      elastic:
    ports:
      - 9200:9200/tcp
    volumes:
        - ./elasticsearch:/usr/share/elasticsearch/data
        - ./certs:/usr/share/elasticsearch/config/certificates
    environment:
      ES_JAVA_OPTS: '-Xms2g -Xmx2g'
      ELASTIC_PASSWORD: 'CHANGEME'
      cluster.name: elastiflow
      node.name: elasticsearch
      bootstrap.memory_lock: 'true'
      network.host: 0.0.0.0
      http.port: 9200
      discovery.type: 'single-node'
      indices.query.bool.max_clause_count: 8192
      search.max_buckets: 250000
      action.destructive_requires_name: 'true'
      reindex.remote.whitelist: '*:*'
      reindex.ssl.verification_mode: 'none'
      xpack.security.http.ssl.enabled: 'false'
      xpack.monitoring.collection.enabled: 'true'
      xpack.monitoring.collection.interval: 30s
      xpack.security.enabled: 'true'
      xpack.security.audit.enabled: 'false'
  # ElastiFlow Unified Flow Collector
  flow-collector:
    image: elastiflow/flow-collector:5.3.5
    container_name: flow-collector
    restart: 'unless-stopped'
    network_mode: 'host'
    volumes:
      - /etc/elastiflow:/etc/elastiflow
    environment:
      EF_FLOW_SERVER_UDP_IP: '0.0.0.0'
      EF_FLOW_SERVER_UDP_PORT: 9995
        
      EF_FLOW_DECODER_ENRICH_IPADDR_METADATA_ENABLE: 'false'
      EF_FLOW_DECODER_ENRICH_DNS_ENABLE: 'true'
      EF_FLOW_DECODER_ENRICH_DNS_NAMESERVER_IP: '1.1.1.1'
      EF_FLOW_DECODER_ENRICH_DNS_NAMESERVER_TIMEOUT: 3000
      EF_FLOW_DECODER_ENRICH_MAXMIND_ASN_ENABLE: 'false'
      EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_ENABLE: 'false'
      EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_ENABLE: 'false'
      EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_ENABLE: 'false
      # Elasticsearch
      EF_FLOW_OUTPUT_ELASTICSEARCH_ENABLE: 'true'
      EF_FLOW_OUTPUT_ELASTICSEARCH_ECS_ENABLE: 'false'
      EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SHARDS: 1
      EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REPLICAS: 0
      # A comma separated list of Elasticsearch nodes to use. DO NOT include "http://" or "https://"
      EF_FLOW_OUTPUT_ELASTICSEARCH_ADDRESSES: '127.0.0.1:9200'
      EF_FLOW_OUTPUT_ELASTICSEARCH_USERNAME: 'kibana_system'
      EF_FLOW_OUTPUT_ELASTICSEARCH_PASSWORD: 'CHANGEME'
      EF_FLOW_OUTPUT_ELASTICSEARCH_TLS_ENABLE: 'false'
      EF_FLOW_OUTPUT_ELASTICSEARCH_TLS_SKIP_VERIFICATION: 'false'
      EF_FLOW_OUTPUT_ELASTICSEARCH_TLS_CA_CERT_FILEPATH: ''
      # Splunk
      EF_FLOW_OUTPUT_SPLUNK_HEC_ENABLE: 'false'
      EF_FLOW_OUTPUT_SPLUNK_HEC_ADDRESSES: '127.0.0.1:8088'
      EF_FLOW_OUTPUT_SPLUNK_HEC_TOKEN: ''
      # Logz.io
      EF_FLOW_OUTPUT_LOGZIO_ENABLE: 'false'
      EF_FLOW_OUTPUT_LOGZIO_ADDRESSES: 'listener.logz.io:8070'
      EF_FLOW_OUTPUT_LOGZIO_TOKEN: ''
      # Kafka
      EF_FLOW_OUTPUT_KAFKA_ENABLE: 'false'
      EF_FLOW_OUTPUT_KAFKA_BROKERS: ''
      #EF_FLOW_OUTPUT_KAFKA_VERSION: '1.0.0'
      #EF_FLOW_OUTPUT_KAFKA_TOPIC: 'elastiflow-flow-codex'
      #EF_FLOW_OUTPUT_KAFKA_PARTITION_KEY: 'flow.export.ip.addr'
      #EF_FLOW_OUTPUT_KAFKA_CLIENT_ID: 'elastiflow-flowcoll'
      #EF_FLOW_OUTPUT_KAFKA_RACK_ID: ''
      #EF_FLOW_OUTPUT_KAFKA_TIMEOUT: 30
      EF_FLOW_OUTPUT_KAFKA_SASL_ENABLE: 'false'
      # Cribl
      EF_FLOW_OUTPUT_CRIBL_ENABLE: 'false'
      EF_FLOW_OUTPUT_CRIBL_ADDRESSES: '127.0.0.1:10080'
      EF_FLOW_OUTPUT_CRIBL_TOKEN: ''
      
      # RiskIQ
      EF_FLOW_OUTPUT_RISKIQ_ENABLE: 'false'
networks:
  elastic:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.enable_icc: "true"
  proxy_network:
    external: true
Code language: YAML (yaml)

ちゃんと動くcompose

んであれこれ悩んで、kibanaとelasticsearchの環境変数をgithubのほうで公開されてるdocker-composeと同じにしたらとりあえず動いた。んで紆余曲折あって最終的に以下のような形で落ち着いた。上記コードはtratailで質問するためにコメント削ってたけど下記は削ってないので読みづらくても我慢してね。

version: '3'
services:
  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.1
    restart: unless-stopped
    hostname: kibana
    # network_mode: bridge
    # volumes:
    #   - /etc/localtime:/etc/localtime:ro
    networks:
      elastic:
      proxy_network:
    ports:
      # HTTP/REST
      - 5601/tcp

    environment:
    #   TZ: Asia/Tokyo
      VIRTUAL_HOST: flow.hoge.jp
      VIRTUAL_PORT: 5601
      I18N_LOCALE: ja-JP
      SERVER_HOST: 0.0.0.0
      SERVER_PORT: 5601
      SERVER_MAXPAYLOADBYTES: 8388608

      ELASTICSEARCH_HOSTS: "http://elasticsearch:9200"
      ELASTICSEARCH_REQUESTTIMEOUT: 132000
      ELASTICSEARCH_SHARDTIMEOUT: 120000

      KIBANA_DEFAULTAPPID: "dashboard/4a608bc0-3d3e-11eb-bc2c-c5758316d788"
      KIBANA_AUTOCOMPLETETIMEOUT: 3000
      KIBANA_AUTOCOMPLETETERMINATEAFTER: 2500000

      LOGGING_DEST: stdout
      LOGGING_QUIET: 'false'

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1
    container_name: elasticsearch
    restart: unless-stopped
    hostname: elasticsearch
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 131072
        hard: 131072
      nproc: 8192
      fsize: -1
    # network_mode: bridge
    networks:
      elastic:
    ports:
      # HTTP/REST
      - 9200:9200/tcp
    volumes:
      # mkdir /var/lib/elasticsearch && chown -R 1000:1000 /var/lib/elasticsearch
      # - /var/lib/elasticsearch:/usr/share/elasticsearch/data
      # - /etc/certs:/usr/share/elasticsearch/config/certificates
        - ./elasticsearch:/usr/share/elasticsearch/data
        - ./certs:/usr/share/elasticsearch/config/certificates
    environment:
      # JVM Heap size
      #   - this should be at least 2GB for simple testing, receiving only a few flows per second.
      #   - for production environments upto 31GB is recommended.
      ES_JAVA_OPTS: '-Xms1g -Xmx1g'

      cluster.name: elastiflow

      bootstrap.memory_lock: 'true'

      network.host: 0.0.0.0
      http.port: 9200
      discovery.type: 'single-node'

      indices.query.bool.max_clause_count: 8192
      search.max_buckets: 250000

      action.destructive_requires_name: 'true'
  # ElastiFlow Unified Flow Collector
  flow-collector:
    image: elastiflow/flow-collector:5.3.5
    container_name: flow-collector
    restart: 'unless-stopped'
    network_mode: 'host'
    volumes:
      - ./flow-collector/:/etc/elastiflow
    depends_on: 
      - elasticsearch
    environment:
      TZ: Asia/Tokyo
      #EF_FLOW_ACCOUNT_ID: ''
      #EF_FLOW_LICENSE_KEY: ''
      #EF_FLOW_LICENSED_CORES: 

      #EF_FLOW_LOGGER_LEVEL: 'info'
      #EF_FLOW_LOGGER_ENCODING: 'json'
      #EF_FLOW_LOGGER_FILE_LOG_ENABLE: 'false'
      #EF_FLOW_LOGGER_FILE_LOG_FILENAME: '/var/log/elastiflow/flowcoll/flowcoll.log'
      #EF_FLOW_LOGGER_FILE_LOG_MAX_SIZE: 100
      #EF_FLOW_LOGGER_FILE_LOG_MAX_AGE: ''
      #EF_FLOW_LOGGER_FILE_LOG_MAX_BACKUPS: 4
      #EF_FLOW_LOGGER_FILE_LOG_COMPRESS: 'false'

      EF_FLOW_SERVER_UDP_IP: '0.0.0.0'
      EF_FLOW_SERVER_UDP_PORT: 6343 # 9995
      #EF_FLOW_SERVER_UDP_PACKET_STREAM_MAX_SIZE: 
      #EF_FLOW_SERVER_UDP_READ_BUFFER_MAX_SIZE: 134217728

      #EF_FLOW_DECODER_SETTINGS_PATH: '/etc/elastiflow'

      #EF_FLOW_DECODER_IPFIX_ENABLE: 'true'
      #EF_FLOW_DECODER_NETFLOW1_ENABLE: 'true'
      #EF_FLOW_DECODER_NETFLOW5_ENABLE: 'true'
      #EF_FLOW_DECODER_NETFLOW6_ENABLE: 'true'
      #EF_FLOW_DECODER_NETFLOW7_ENABLE: 'true'
      #EF_FLOW_DECODER_NETFLOW9_ENABLE: 'true'
      #EF_FLOW_DECODER_SFLOW5_ENABLE: 'true'
      #EF_FLOW_DECODER_SFLOW_FLOWS_ENABLE: 'true'
      #EF_FLOW_DECODER_SFLOW_FLOWS_KEEP_SAMPLES: 'false'
      #EF_FLOW_DECODER_SFLOW_COUNTERS_ENABLE: 'true'

      #EF_FLOW_DECODER_TRANSLATE_KEEP_IDS: 'default'

      #EF_FLOW_DECODER_ENRICH_IPADDR_QUEUE_SIZE: 65536
      #EF_FLOW_DECODER_ENRICH_IPADDR_TTL: 7200
      #EF_FLOW_DECODER_ENRICH_IPADDR_CUSTODIAN_PASS_DELAY: 60000
        
      EF_FLOW_DECODER_ENRICH_IPADDR_METADATA_ENABLE: 'false'
      #EF_FLOW_DECODER_ENRICH_IPADDR_METADATA_USERDEF_PATH: 'metadata/ipaddrs.yml'
      #EF_FLOW_DECODER_ENRICH_IPADDR_METADATA_REFRESH_RATE: 15

      EF_FLOW_DECODER_ENRICH_DNS_ENABLE: 'true'
      EF_FLOW_DECODER_ENRICH_DNS_NAMESERVER_IP: '1.1.1.1'
      EF_FLOW_DECODER_ENRICH_DNS_NAMESERVER_TIMEOUT: 3000
      #EF_FLOW_DECODER_ENRICH_DNS_RESOLVE_PRIVATE: 'true'
      #EF_FLOW_DECODER_ENRICH_DNS_RESOLVE_PUBLIC: 'true'
      #EF_FLOW_DECODER_ENRICH_DNS_USERDEF_PATH: 'hostname/user_defined.yml'
      #EF_FLOW_DECODER_ENRICH_DNS_USERDEF_REFRESH_RATE: 15
      #EF_FLOW_DECODER_ENRICH_DNS_INCLEXCL_PATH: 'hostname/incl_excl.yml'
      #EF_FLOW_DECODER_ENRICH_DNS_INCLEXCL_REFRESH_RATE: 15

      EF_FLOW_DECODER_ENRICH_MAXMIND_ASN_ENABLE: 'true'
      #EF_FLOW_DECODER_ENRICH_MAXMIND_ASN_PATH: 'maxmind/GeoLite2-ASN.mmdb'

      EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_ENABLE: 'true'
      #EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_PATH: 'maxmind/GeoLite2-City.mmdb'
      #EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_VALUES: 'city,country,country_code,location,timezone'
      EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_LANG: 'ja'
      #EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_INCLEXCL_PATH: 'maxmind/incl_excl.yml'
      #EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_INCLEXCL_REFRESH_RATE: 15

      EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_ENABLE: 'true'
      #EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_ENDPOINT: 'https://api.passivetotal.org/v2/netflow/as/download'
      #EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_REFRESH_INTERVAL: 1440
      EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_ENABLE: 'true'
      #EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_ENDPOINT: 'https://api.passivetotal.org/v2/netflow/blocklist/download'
      #EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_REFRESH_INTERVAL: 1440
      #EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_INCLEXCL_PATH: 'riskiq/incl_excl.yml'
      #EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_INCLEXCL_REFRESH_RATE: 15
      EF_FLOW_DECODER_ENRICH_RISKIQ_API_USER: 'example@hoge.com'
      EF_FLOW_DECODER_ENRICH_RISKIQ_API_KEY: '16進数の羅列みたいな文字列'
      #EF_FLOW_DECODER_ENRICH_RISKIQ_API_TIMEOUT: 180

      #EF_FLOW_DECODER_ENRICH_ASN_PREF: 'lookup'

      #EF_FLOW_DECODER_ENRICH_NETIF_GET_ATTRS: 'true'
      #EF_FLOW_DECODER_ENRICH_NETIF_CACHE_SIZE: 262144

      #EF_FLOW_DECODER_ENRICH_SNMP_ENABLE: 'false'
      #EF_FLOW_DECODER_ENRICH_SNMP_PORT: 161
      #EF_FLOW_DECODER_ENRICH_SNMP_VERSION: 2
      #EF_FLOW_DECODER_ENRICH_SNMP_COMMUNITY: 'public'
      #EF_FLOW_DECODER_ENRICH_SNMP_TIMEOUT: 2
      #EF_FLOW_DECODER_ENRICH_SNMP_RETRIES: 1

      #EF_FLOW_DECODER_ENRICH_APP_CACHE_SIZE: 8388608
      
      #EF_FLOW_DECODER_ENRICH_APP_USERDEF_ENABLE: 'false'
      #EF_FLOW_DECODER_ENRICH_APP_USERDEF_PRIVATE: 'true'
      #EF_FLOW_DECODER_ENRICH_APP_USERDEF_PUBLIC: 'true'
      #EF_FLOW_DECODER_ENRICH_APP_USERDEF_PATH: 'settings/apps_user_defined.yml'

      #EF_FLOW_DECODER_ENRICH_TOTALS_IF_NO_DELTAS: 'false'

      #EF_FLOW_DECODER_ENRICH_SAMPLERATE_CACHE_SIZE: 32768
      #EF_FLOW_DECODER_ENRICH_SAMPLERATE_USERDEF_ENABLE: 'false'
      #EF_FLOW_DECODER_ENRICH_SAMPLERATE_USERDEF_PATH: 'settings/sample_rate.yml'

      #EF_FLOW_DECODER_ENRICH_COMMUNITYID_ENABLE: 'true'
      #EF_FLOW_DECODER_ENRICH_COMMUNITYID_SEED: 0
      #EF_FLOW_DECODER_ENRICH_CONVERSATIONID_ENABLE: 'true'
      #EF_FLOW_DECODER_ENRICH_CONVERSATIONID_SEED: 0

      #EF_FLOW_DECODER_ENRICH_JOIN_ASN: 'true'
      #EF_FLOW_DECODER_ENRICH_JOIN_GEOIP: 'true'
      #EF_FLOW_DECODER_ENRICH_JOIN_SEC: 'true'
      #EF_FLOW_DECODER_ENRICH_JOIN_NETATTR: 'true'
      #EF_FLOW_DECODER_ENRICH_JOIN_SUBNETATTR: 'true'

      #EF_FLOW_DECODER_DURATION_PRECISION: 'ms'
      #EF_FLOW_DECODER_TIMESTAMP_PRECISION: 'ms'
      #EF_FLOW_DECODER_PERCENT_NORM: 100
      #EF_FLOW_DECODER_ENRICH_EXPAND_CLISRV: 'true'
      #EF_FLOW_DECODER_ENRICH_KEEP_CPU_TICKS: 'false'

      #EF_FLOW_DECODER_ENRICH_DROP_FIELDS: ''

      #EF_FLOW_RECORD_STREAM_MAX_SIZE: 

      # stdout
      #EF_FLOW_OUTPUT_STDOUT_ENABLE: 'false'
      #EF_FLOW_OUTPUT_STDOUT_FORMAT: 'json_pretty'

      # monitor
      #EF_FLOW_OUTPUT_MONITOR_ENABLE: 'false'
      #EF_FLOW_OUTPUT_MONITOR_INTERVAL: 300

      # Elasticsearch
      EF_FLOW_OUTPUT_ELASTICSEARCH_ENABLE: 'true'
      EF_FLOW_OUTPUT_ELASTICSEARCH_ECS_ENABLE: 'false'
      #EF_FLOW_OUTPUT_ELASTICSEARCH_BATCH_DEADLINE: 2000
      #EF_FLOW_OUTPUT_ELASTICSEARCH_BATCH_MAX_BYTES: 8388608
      #EF_FLOW_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE: 'end'
      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_PERIOD: 'daily'
      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_SUFFIX: ''

      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ENABLE: 'true'
      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_OVERWRITE: 'true'
      EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SHARDS: 1
      EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REPLICAS: 0
      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REFRESH_INTERVAL: '10s'
      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_CODEC: 'best_compression'
      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ILM_LIFECYCLE: ''
      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ILM_ROLLOVER_ALIAS: ''
      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ISM_POLICY: ''
      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_PIPELINE_DEFAULT: '_none'
      #EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL: '_none'

      # A comma separated list of Elasticsearch nodes to use. DO NOT include "http://" or "https://"
      EF_FLOW_OUTPUT_ELASTICSEARCH_ADDRESSES: '127.0.0.1:9200'
      # EF_FLOW_OUTPUT_ELASTICSEARCH_USERNAME: 'kibana_system'
      # EF_FLOW_OUTPUT_ELASTICSEARCH_PASSWORD: 'CHANGEME'
      #EF_FLOW_OUTPUT_ELASTICSEARCH_CLOUD_ID: ''
      #EF_FLOW_OUTPUT_ELASTICSEARCH_API_KEY: ''

      EF_FLOW_OUTPUT_ELASTICSEARCH_TLS_ENABLE: 'false'
      EF_FLOW_OUTPUT_ELASTICSEARCH_TLS_SKIP_VERIFICATION: 'false'
      EF_FLOW_OUTPUT_ELASTICSEARCH_TLS_CA_CERT_FILEPATH: ''

      EF_FLOW_OUTPUT_ELASTICSEARCH_RETRY_ENABLE: 'true'
      EF_FLOW_OUTPUT_ELASTICSEARCH_RETRY_ON_TIMEOUT_ENABLE: 'true'
      EF_FLOW_OUTPUT_ELASTICSEARCH_MAX_RETRIES: 10
      EF_FLOW_OUTPUT_ELASTICSEARCH_RETRY_BACKOFF: 10000 #デフォは1000ms。elasticsearch待ち 

      # Splunk
      EF_FLOW_OUTPUT_SPLUNK_HEC_ENABLE: 'false'
      #EF_FLOW_OUTPUT_SPLUNK_HEC_CIM_ENABLE: 'false'
      EF_FLOW_OUTPUT_SPLUNK_HEC_ADDRESSES: '127.0.0.1:8088'
      EF_FLOW_OUTPUT_SPLUNK_HEC_TOKEN: ''
      #EF_FLOW_OUTPUT_SPLUNK_HEC_BATCH_MAX_BYTES: 8388608
      #EF_FLOW_OUTPUT_SPLUNK_HEC_BATCH_DEADLINE: 2000
      #EF_FLOW_OUTPUT_SPLUNK_HEC_TLS_ENABLE: 'true'
      #EF_FLOW_OUTPUT_SPLUNK_HEC_TLS_SKIP_VERIFICATION: 'false'
      #EF_FLOW_OUTPUT_SPLUNK_HEC_TLS_CA_CERT_FILEPATH: ''

      # Logz.io
      EF_FLOW_OUTPUT_LOGZIO_ENABLE: 'false'
      EF_FLOW_OUTPUT_LOGZIO_ADDRESSES: 'listener.logz.io:8070'
      EF_FLOW_OUTPUT_LOGZIO_TOKEN: ''
      #EF_FLOW_OUTPUT_LOGZIO_TIMESTAMP_SOURCE: 'end'
      #EF_FLOW_OUTPUT_LOGZIO_BATCH_DEADLINE: 2000
      #EF_FLOW_OUTPUT_LOGZIO_BATCH_MAX_BYTES: 8388608
      #EF_FLOW_OUTPUT_LOGZIO_ECS_ENABLE: 'false'
      #EF_FLOW_OUTPUT_LOGZIO_TIMEOUT: 30000
      #EF_FLOW_OUTPUT_LOGZIO_TLS_ENABLE: 'false'

      # Kafka
      EF_FLOW_OUTPUT_KAFKA_ENABLE: 'false'
      EF_FLOW_OUTPUT_KAFKA_BROKERS: ''
      #EF_FLOW_OUTPUT_KAFKA_VERSION: '1.0.0'
      #EF_FLOW_OUTPUT_KAFKA_TOPIC: 'elastiflow-flow-codex'
      #EF_FLOW_OUTPUT_KAFKA_PARTITION_KEY: 'flow.export.ip.addr'
      #EF_FLOW_OUTPUT_KAFKA_CLIENT_ID: 'elastiflow-flowcoll'
      #EF_FLOW_OUTPUT_KAFKA_RACK_ID: ''
      #EF_FLOW_OUTPUT_KAFKA_TIMEOUT: 30

      EF_FLOW_OUTPUT_KAFKA_SASL_ENABLE: 'false'
      #EF_FLOW_OUTPUT_KAFKA_SASL_USERNAME: ''
      #EF_FLOW_OUTPUT_KAFKA_SASL_PASSWORD: ''

      #EF_FLOW_OUTPUT_KAFKA_TLS_ENABLE: 'false'
      #EF_FLOW_OUTPUT_KAFKA_TLS_CA_CERT_FILEPATH: ''
      #EF_FLOW_OUTPUT_KAFKA_TLS_CERT_FILEPATH: ''
      #EF_FLOW_OUTPUT_KAFKA_TLS_KEY_FILEPATH: ''
      #EF_FLOW_OUTPUT_KAFKA_TLS_SKIP_VERIFICATION: 'false'

      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_MAX_MESSAGE_BYTES: 1000000
      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_REQUIRED_ACKS: 1
      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_TIMEOUT: 10
      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_COMPRESSION: 0
      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_COMPRESSION_LEVEL: -1000
      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_FLUSH_BYTES: 1000000
      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_FLUSH_MESSAGES: 1024
      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_FLUSH_FREQUENCY: 500
      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_FLUSH_MAX_MESSAGES: 0
      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_RETRY_MAX: 3
      #EF_FLOW_OUTPUT_KAFKA_PRODUCER_RETRY_BACKOFF: 100

      # Cribl
      EF_FLOW_OUTPUT_CRIBL_ENABLE: 'false'
      EF_FLOW_OUTPUT_CRIBL_ADDRESSES: '127.0.0.1:10080'
      EF_FLOW_OUTPUT_CRIBL_TOKEN: ''
      #EF_FLOW_OUTPUT_CRIBL_BATCH_DEADLINE: 2000
      #EF_FLOW_OUTPUT_CRIBL_BATCH_MAX_BYTES: 8388608
      #EF_FLOW_OUTPUT_CRIBL_TLS_ENABLE: 'false'
      #EF_FLOW_OUTPUT_CRIBL_TLS_SKIP_VERIFICATION: 'false'
      #EF_FLOW_OUTPUT_CRIBL_TLS_CA_CERT_FILEPATH: ''
      
      #RiskIQ
      EF_FLOW_OUTPUT_RISKIQ_ENABLE: 'true'
      EF_FLOW_OUTPUT_RISKIQ_HOST: 'flow.riskiq.net'
      EF_FLOW_OUTPUT_RISKIQ_PORT: 20000
      EF_FLOW_OUTPUT_RISKIQ_CUSTOMER_UUID: 'UUIDの文字列'
      EF_FLOW_OUTPUT_RISKIQ_CUSTOMER_ENCRYPTION_KEY: 'keyの文字列'
networks:
  elastic:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.enable_icc: "true"
  proxy_network:
    external: true
Code language: YAML (yaml)

GeoIP関連

上記composeそのままupしても動かない項目なので説明しとく。flow-collectorのvolumesで./flow-collector/:/etc/elastiflowみたいなマウントしてるけど、中身には以下のようなファイルをおいてる。

flow-collector
└ maxmind
├ GeoLite2-ASN.mmdb
└ GeoLite2-City.mmdb

ようするに以下のような画面で送信元や送信先のロケーション確認できる機能を使えるようにするためのdbファイルね。このファイルを設置してGeoIP関連の環境変数で有効にしないと動かない。ファイル入手先の情報やら関連する環境変数の説明は公式ページ参照。

GeoIP

RiskIQ関連

こんな感じのセキュリティリスク可視化する的な機能を有効化するやつね。GeoIPと同じくそのままだと動かない機能。RiskIQってやつのアカウント取得してAPI使えるようにして環境変数を適切に設定してあげたら動く。これもアカウント取得手順とか設定する環境変数とかは公式ページのドキュメント参照。

proxy_network

わいがkibanaにバーチャルホスト名でアクセスするためにリバースプロキシ経由用で書いてる。リバースプロキシはさみたくない人は直接kibanaのコンテナのport開けて使えばええと思う。

upした後

上記composeをupしたら公式ページから自分のkibanaのバージョンに合うダッシュボードのファイルをDLしてkibana上でインポートする細かい手順は公式ページのドキュメント読んでね。自分はkibanaを7.17で構築してるので一番上を選んだ。

あとはsflowとかnetflow対応してるネットワーク機器から動かしてるサーバ宛にエクスポートしたら動くんじゃね。

Category: 電子情報工学 | Tag: ,