Análise de Dados

Programming skills (variables, control structures, functions and basic object-oriented programming), knowledge of descriptive statistics and probability, and knowledge of relational databases (relational model and SQL).

The course unit combines theoretical and practical approaches so that students acquire both the conceptual knowledge and the technical skills required for data analysis in industrial contexts. The CRISP-DM methodology serves as the guiding framework for the knowledge discovery component.

1. Lectures

In the lecture segments of the theoretical-practical classes, fundamental concepts are presented in a structured way and illustrated with industrial cases and problems.

2. Laboratory work and problem solving

Practical segments take place in the computer lab, using Python data analysis libraries and database and SQL tools. Students solve exercises that apply and consolidate the techniques studied, focusing on:

• Exploring and preparing real datasets, including process and production data.
• Applying classification, regression and clustering models.
• Critically evaluating models and interpreting results.
• Multidimensional modelling and developing ETL processes.

3. Project-based and collaborative learning

In small groups, students develop a project covering the full data analysis cycle, applied to an industrial problem. The project is supervised during classes and concludes with a written report and an oral presentation, followed by a defence. Guided discussion of each group’s results and methodological choices fosters critical analysis and knowledge sharing.

Objectives

This course unit aims to:

• Understand and apply data exploration, preparation and analysis techniques for knowledge discovery.
• Become familiar with the principles of data warehousing and multidimensional modelling.
• Design processes for extracting, cleaning and transforming data from different sources.

Skills

By the end of the course unit, students should be able to:

• Explore and prepare datasets, particularly data from industrial contexts.
• Apply and evaluate simple classification, regression and clustering models, following the CRISP-DM methodology.
• Design a simple multidimensional model (e.g. a star schema) for a data warehouse.
• Implement a basic ETL (extract, transform, load) process

1. Knowledge Discovery from Data (Data Mining)

1.1 Introduction to data-driven project methodology: concepts and objectives; the CRISP-DM methodology; types of problems in industrial settings.

1.2 Data understanding: industrial data sources (sensors, PLC/SCADA, MES, ERP); descriptive statistics, distributions and correlations; visualisation; data quality assessment.

1.3 Data preparation: missing values and outliers; normalisation and encoding of variables; feature engineering and selection; introduction to time series processing.

1.4 Supervised and unsupervised models: classification, regression and clustering; criteria for choosing the type of model according to the problem.

1.5 Learning techniques: decision trees, k-NN and Naïve Bayes; linear regression; k-means and hierarchical clustering; introduction to anomaly detection.

1.6 Evaluation and validation metrics: training, validation and test sets; cross-validation; overfitting and underfitting; classification metrics (confusion matrix, accuracy, precision, recall/sensitivity, F1 and AUC-ROC); regression metrics (MAE and RMSE).

2. Data Warehousing and Multidimensional Modelling

2.1 Introduction to data warehousing: transactional (OLTP) and analytical (OLAP) systems; the role of the data warehouse in decision support.

2.2 Data warehouse architecture: data sources, staging area, data warehouse and data marts.

2.3 Multidimensional model concepts: facts, dimensions and measures; granularity; hierarchies; OLAP operations (roll-up, drill-down, slice and dice).

2.4 Multidimensional schemas: star schema and snowflake schema; comparison.

2.5 Data extraction and transformation (ETL): extraction from multiple sources; data cleaning, integration and transformation.

2.6 Data loading (ETL): initial and incremental loads; load frequency.

The final grade comprises an individual component (60%, 12 points) and a group project (40%, 8 points), on a scale of 0 to 20. Students may choose to complete the individual component through continuous assessment or by exam.

Continuous assessment (60%, 12 points). Consists of problems set in class throughout the semester, on dates not announced in advance. A minimum grade of 35% (4.2 points) is required.

Exam (60%, 12 points). Covers all programme contents, with a minimum required grade of 35% (4.2 points). The exam is open to students who did not choose continuous assessment or who did not reach the minimum grade in it, without prejudice to other situations provided for in the regulations in force.

Project (40%, 8 points). Carried out in groups, the project consists of applying the full data analysis cycle to an industrial problem, with submission of a written report and an oral presentation, followed by a defence. Grades may differ among group members according to individual performance in the defence. A minimum grade of 35% (2.8 points) is required. The grade obtained remains valid for all assessment periods of the academic year.

Final grade. The sum of the continuous assessment or exam grade and the project grade. Students pass with a final grade of 9.5 points or higher, provided the minimum grade in each component is met.

Additional provisions. The course unit coordinator may supplement the assessment with an oral examination whenever justified, namely to confirm the authorship of submitted work. Students with a grade of 17 points or higher must take an oral examination. In both cases, the oral examination is notified at least 48 hours in advance and may result in the grade being maintained, raised or lowered; failure to attend results in failing the course unit.

NAO