AiVibeData Lake
A data lake is a central repository that stores large volumes of raw structured, semi-structured and unstructured data in its native format, typically on low-cost object storage, for later analytics and machine learning.
A data lake is a centralised repository that stores large volumes of raw data in its native format, whether structured tables, semi-structured JSON and logs, or unstructured images and documents. Data is usually kept in low-cost object storage, such as Amazon S3, Azure Data Lake Storage or Google Cloud Storage, often in open file formats like Parquet. Unlike a data warehouse, a data lake applies schema on read: structure is imposed when data is queried or processed, not when it is loaded.
Data lakes support data science, machine learning and exploratory analytics that need full-fidelity history, such as years of sensor readings, machine logs, inspection images and quality records. Manufacturers use them to combine plant data with business data for analyses such as yield improvement and training predictive maintenance models. Query engines such as Apache Spark, Trino and Amazon Athena process the data in place.
Without governance, a data lake can become a so-called data swamp of undocumented, untrusted and duplicated files. Essential practices include a data catalogue with metadata and lineage, consistent naming and partitioning, access control down to sensitive columns, data quality checks and retention rules. Many organisations now add open table formats to their lakes, creating the lakehouse pattern with transactional updates and better query performance.
Key points
- Stores raw data in native formats, usually on cloud object storage
- Uses schema on read, unlike the schema-on-write approach of data warehouses
- Suits data science, machine learning and long-term sensor and log history
- Needs a catalogue, lineage and access control to avoid becoming a data swamp
Where AiVibe comes in
AiVibe Software Services delivers cloud solutions on AWS, Microsoft Azure, Google Cloud or on-premise, together with cloud security, legacy modernisation, data analytics and AI and machine learning services.
Related terms
- Data WarehouseCloud & AI Infrastructure
- Data LakehouseCloud & AI Infrastructure
- Object StorageCloud & AI Infrastructure
- Extract, Transform, Load (ETL)Cloud & AI Infrastructure
- Predictive Maintenance (PdM)Quality, Reliability & Maintenance