Artificial Intelligence (AI): Machine learning (ML), Deep Learning (DP); оценка эффективности ML моделей

accuracy = (tp + tn) / (tp + fp + fn + tn)
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1_score = 2 * (recall * precision) / (recall + precision)
    • accuracy, лично я считаю, является наиболее универсальной и главное понятной метрикой оценки, если пытаться осуществлять оценку только на основе одной метрики (attention: автор на практике в статье выше описал почему при сравнении моделей это плохое упрощение) – это простая сумма всех правильных решений модели/сумма всех решений. Кроме того остальные метрики/показатели сознательно не учитывают верно-отрицательные (TN) решения и это, очевидно, спорно.

Artificial Intelligence (AI) is a category of computing that adapts and improves its decision-making ability over time based on its successes and failures.

  • AI, in the context of cloud computing, is based around a broad range of services, the core of which is machine learning.
  • Machine learning is a data science technique that allows computers to use existing data to forecast future behaviors, outcomes, and trends.
  • Using machine learning, computers learn without being explicitly programmed.
  • AI is one focus that could transform every area of a business. Such transformation is limited only by the creativity and imagination of the organization.
  • Forecasts or predictions from machine learning can make apps and devices smarter. For example, when you shop online, machine learning helps recommend other products you might like based on what you’ve purchased.
The Marketing team is convinced that it can increase sales dramatically by suggesting add-on products that complement the items in a shopper's cart at the point of checkout. 
The team could hard-code these suggestions, but it feels that a more organic approach would be to use its years' worth of sales data as well as new shopping trends to decide what products to display to the shopper. 
Additionally, the suggestions could be influenced by product availability, product profitability, and other factors.
The Marketing team's existing data science experts have already done some initial analysis of the problem domain, and have determined that its plan might take months to prototype, and possibly a year to roll out.
> Finally, it sounds like the Marketing team already employs some data science experts, and the team is willing to make at least a year-long commitment to building, testing, and tweaking the models to be used.

there are two basic approaches to AI

  • The first is to employ a deep learning system that’s modeled on the neural network of the human mind, enabling it to discover, learn, and grow through experience.
  • The second approach is machine learning, a data science technique that uses existing data to train a model, test it, and then apply the model to new data to forecast future behaviors, outcomes, and trends.


machine learning (ML)
  • О DGA detection с помощью ML подробнее в DNS, из интересного: No machine learning model is perfect! Some benign domains will be mistakenly labeled as false positives.
  • Пример огромного количества эвентов, которые может генерировать SIEM с ML аналитикой (в данном случае Elastic Stack Detection Rules) на основе пользовательской активности (со знанием что норма, а что нет для каждого пользователя) – процессы, сеть, ааа, геолокация, время доступа и проч.
  • Пример почему ML зачастую лучше чем
    • сигнатуры – сигнатуры нужно писать, а ML уже “как-то” обучен и работает. Кроме того сигнатуры нужно писать универсально, что в некоторых случаях невозможно (пример с DGA).
    • ревер инжениринг – нужно разбираться как каждый из DGA алгоритмов и вариаций конфигураций этих алгоритмов работает перед тем, как блокировать запросы к доменам

Virtually every device or software system that collects textual, visual, and audio data could feed a machine learning model that makes that device or software system smarter about how it functions in the future.

Forecasts or predictions from machine learning can make apps and devices smarter. For example, when you shop online, machine learning powers product recommendation systems that offer additional products based on what you’ve bought and what other shoppers have bought who have purchased similar items in the past. Machine learning is also used to detect credit card fraud by analyzing each new transaction and using what it has learned from analyzing millions of fraudulent transactions.

Machine learning deployment steps:
1 – define a goal for the ML system
2 – acquire sample data containing factors that might correlate to a positive or negative decision.
Analyse the data to ensure there were no biases. That the sample represents the entire population of data. And that there’s no order or weight implied.
The quantity and the quality of the data is vitally important, since it will be what is used to train and evaluate the system.
The data must contain the answer correlated to our goal.
3 – split data into:
– training data for building the algorithm (the rules for making new decisions based on similar data in the future)
– the evaluation data to test the algorithm
4 – choose a model (data scientist & researchers created models for different purposes: some work well with visual data, others with sequence data/text-based data) these models will be used to generate algorithm
5 – train model supplying our training data and allowing our model to generate an algorithm that correlated various factors into a decision. The model decides which factors should be weighted and under what circumstances.
6 – use the evaluation data to test our new algorithm to see how accurate it is (example: 95%)
7 – after testing we may need to tweak the algorithm by hand tuning certain parameters and retesting
8 – deploy the system into a live environment where its conclusions can be utilized by our business

As we use the system, we can use our results to continue to train the model.


часть задач эксплуатационных, часть аналитических (опенца моделей0

         – для нас достаточно accuracy (сумма верных решений/сумма всех решений) – увидим (если она есть) деградацию под нагрузкой с учетом всех показателей (TP, TN, FP, FN) при тестировании одним набором данных от итерации к итерации

         – остальные показатели не используем (precision, recall, f1score) – они в первую очередь нужны и используются для сравнительной оценки разных ML моделей между собой перед выбором конкретной модели

Leave a Reply