Description

Introduction

This training provides a comprehensive, hands-on introduction to working with core data structures in q, the query language of kdb+. Tables, lists, and dictionaries form the foundation of data manipulation in q, especially in high-performance and time-series environments.

Participants will learn how to create, transform, query, and optimize these structures efficiently. The course focuses on practical use cases relevant to financial data, real-time analytics, and large-scale data processing, enabling learners to confidently build and manipulate in-memory and on-disk datasets.

By the end of the training, participants will understand how q’s columnar architecture and vector-based operations deliver powerful performance advantages.

Prerequisites

1. Basic understanding of programming concepts (variables, functions, data types)
2. Familiarity with database fundamentals (rows, columns, keys, joins)
3. Introductory knowledge of q syntax
4. Basic understanding of time-series data concepts (helpful for applied examples)

Table of Contents

Module 1: Foundations of q Data Structures

1. Overview of q data types
2. Atoms vs Lists
3. Type system and type casting
4. Nulls and infinities in q
5. Memory model basics

Module 2: Working with Lists

1. Creating simple and typed lists
2. Numeric, symbol, temporal, and mixed lists
3. Indexing and slicing
4. List operations and vectorized computation
5. Functional operations on lists
6. Performance considerations with large lists

Module 3: Dictionaries in q

1. Creating dictionaries
2. Key-value mapping fundamentals
3. Accessing and updating dictionary values
4. Enumerations and symbol handling
5. Nested dictionaries
6. Using dictionaries in real-world scenarios

Module 4: Introduction to Tables

1. Tables as flipped dictionaries
2. Creating tables manually
3. Loading tables from data
4. Column types and metadata
5. Viewing and inspecting table structure

Module 5: Querying Tables

1. Select queries in q
2. Where clauses and filtering
3. Aggregations and grouping
4. Ordering and sorting
5. Limiting and distinct queries
6. Functional form vs SQL-like form

Module 6: Keyed Tables

1. Creating keyed tables
2. Primary keys and uniqueness
3. Updating keyed tables
4. Upserts and merges
5. Differences between keyed and unkeyed tables

Module 7: Table Transformations

1. Adding and deleting columns
2. Updating rows
3. Joining tables (aj, lj, ij, etc.)
4. Pivoting and reshaping data
5. Working with temporal joins

Module 8: Performance & Best Practices

1. Columnar storage principles
2. In-memory vs on-disk tables
3. Efficient filtering and grouping
4. Avoiding common performance pitfalls
5. Optimizing large datasets

Module 9: Real-World Hands-On Exercises

1. Building a trade & quote table
2. Aggregating market data
3. Creating keyed lookup dictionaries
4. Designing a small in-memory analytics pipeline