Need clarification of the input data schema

[jinxuan]

Is one row represent one tcp connection? If so then why in column 20 ~ 35, we have “% of connections”? If not, does it all belong to one service? How many connection does it represent?

Why is column 1 src_bytes less than 1? What is the unit for it?

What is “compromised conditions” in column 10?

Is it “hot” or “host” in column 7? If it is hot, could you please elaborate more about it?

Need clarification about column 40 ~ 43. What is ECR network service, Private network service. SO status and SF status.

[amityadav]

Greetings!!!
Please see answers to your queries below

Is one row represent one tcp connection?
Answer - Yes
If so then why
in column 20 ~ 35, we have “% of connections”? If not, does it all
belong to one service? How many connection does it represent? ​
Answer - These refer to the percentage of same host and same service connections respectively.
The
"same host" features examine only the connections in the past two
seconds that have the same destination host as the current connection,
and calculate statistics related to protocol behavior, service, etc.
The
similar "same service" features examine only the connections in the
past two seconds that have the same service as the current connection.
"Same host" and "same service" features are together called time-based traffic features of the connection records.
Why is column 1 src_bytes less than 1? What is the unit for it?
Answer
- Unit is bytes. It appears negative as following data normalization is
applied to it: normalized(i) = feature(i) − μ(i)/σ(i)
What is “compromised conditions” in column 10?
Answer - "compromised conditions" is accepting network connection with exceptional network. Value "-0.0021" represent "no"
Is it “hot” or “host” in column 7? If it is hot, could you please elaborate more about it?
Term is "hot". A hot site is an off-premises location where a company's work can resume during a disaster. Values "-0.0265
" represent "in premises".
Need clarification about column 40 ~ 43. What is ECR network service, Private network service. SO status and SF status.
ECR network
service represent "public network", private network service represent
"home network". "SF Status" represent encrypted connectivity. "SO" status represent normal connectivity.

[jinxuan]

Hi,

Thank you for the clarification.

When we analysis the data we're still face some difficulties. Since the data you provided is usually normalized, and the normalization function is not provided. It is hard to interpret the meaning of the data.

I really wish we could see the raw data of those fields. Or provide the function that do the normalization.

For example: for field 19 count: What's the meaning of 0.83 and -1.5754? which one is larger? Which represent 0 connection? Without seeing the raw data it is hard to know that.

Also for field num_root, does 0.0033 means zero root access? Or one? What about 0.0096 for num_file_creations? -0.0085 for num_shells? -0.0288 for num_access_files? 0.0 for is_guest_log_in?

---

For These fields, does 1 represent means Boolean True for all of the following? For example, protocol_type_tcp=1 means the connection is using TCP :

38 protocol_type_icmp
39 protocol_type_tcp
40 service_ecr_i
41 service_private
42 flag_S0
43 flag_SF

I also want to confirm one thing:

Field 9 logged_in: 1 if successfully logged in; 0 otherwise. Is that really true? Based on the data pattern, that means majority of training data set is not successful logged in?

[amityadav]

Hi Jinxuan,

Please find answer below to your queries,

1) Normalization function applied over the data set is already mentioned in previous post i.e. ​

    normalized = (featurei − μi) / σi

here, "μ" is Mean and "σ"​ is Standard Deviation

2) Column numbers 38 - 43 are binary features where:
if protocol_type_icmp has value 1 then protocol_type is “icmp”,
if protocol_type_tcp has value 1 then protocol_type is “tcp”,
when both protocol_type_icmp and protocol_type_tcp has 0 then it implies other protocols.

Similarly,
if service_ecr_i is 1 then service type is “ecr_i”,
if service_private is 1 then service type is “private”,
"Other" services will be captured when both service_ecr_i and service_private has 0.

if flag_S0 is 1 then flag type is “S0”,
if flag_SF is 1 then flag type is “SF”,
"Other" flags will be captured when both flag_S0 and flag_SF has 0.

3) Yes, Field#9 i.e. "logged_in" is also a binary feature where '1' represents user is successfully logged in, '0' otherwise