ETL, Statistics, and Dashboards

This project involved a rigorous statistical analysis of data derived from a random number generator, leveraging custom Python implementations of statistical tests, including the Chi-square (CS), Kolmogorov-Smirnov (KS), and Anderson-Darling (AD) tests, each crafted from foundational principles. The analysis focused on the frequency patterns of 8-bit segment pairs within 64-bit numbers, exploring both an overall distribution approach and a conditional probability approach by segmenting pairs based on the initial segment. Cleaned and meticulously organized data was stored in structured Excel sheets, setting a strong foundation for visual interpretation. Results were conveyed through a sophisticated BI dashboard, developed with Plotly Dash and deployed on Heroku, providing dynamic visual insights through various plots to enhance data accessibility and interpretation.

Deep and Shallow ML

The project tackled the challenge of predicting the distance to the nearest perfect square (either previous or next) for a given number xxx using both shallow and deep supervised machine learning models. The dataset encompassed integers from 1 to 100,000, detailing their closest square numbers along with additional mathematical features such as digital root and polarity. The supervised learning approach included two methodologies: a shallow learning technique using various estimators, with XGBoost emerging as the top performer, and a deep learning approach implemented with the PyTorch library. Results revealed that XGBoost outperformed the deep learning model, achieving notable accuracy, showcasing its suitability for this specific prediction task.