TinyDB - NOSQL database

Complete Guide to TinyDB: Lightweight NoSQL Database for Python

Introduction

Modern applications don’t always need heavyweight, complex DBMS. When a project doesn’t require massive scalability or ultra‑fast processing of huge data sets, a lightweight embedded database is sufficient. TinyDB – a NoSQL database written entirely in Python – offers a convenient and simple solution for storing structured data without SQL.

TinyDB stores data in JSON format and requires no server installation. This makes it ideal for scripts, small CLI tools, desktop applications, prototypes, and educational projects. The library is only a few kilobytes in size and has minimal dependencies, making it an excellent choice for lightweight solutions.

Installation and Basic Setup

Installing TinyDB

You can install TinyDB with a single pip command:

pip install tinydb

Optional extensions add extra features:

pip install tinydb[compression]  # Enable data compression
pip install tinydb[yaml]         # Add YAML support

Creating a Database

Creating a file‑based database:

from tinydb import TinyDB
db = TinyDB('db.json')  # The file will be created automatically

Creating an in‑memory database:

from tinydb.storages import MemoryStorage
db = TinyDB(storage=MemoryStorage)

Encoding and Parameter Settings

from tinydb import TinyDB
from tinydb.storages import JSONStorage

# Set encoding
db = TinyDB('db.json', encoding='utf-8')

# Pretty‑print JSON with indentation
db = TinyDB('db.json', indent=4, separators=(',', ': '))

Fundamentals of TinyDB Data Structure

TinyDB uses a document‑oriented data model:

• Data is stored as a list of documents (Python dictionaries)
• Each document has a unique doc_id generated automatically
• Supported types: strings, numbers, lists, dictionaries, booleans, and null
• Documents in the same table can have different structures
• Nested data of any depth is supported

Core Data Operations

Inserting Data

Insert a single document:

from tinydb import TinyDB
db = TinyDB('db.json')

# Simple insert
db.insert({'name': 'Ivan', 'age': 30, 'city': 'Moscow'})

# Insert with returned ID
doc_id = db.insert({'name': 'Anna', 'age': 25})
print(f"Document added with ID: {doc_id}")

Batch insert:

users = [
    {'name': 'Anna', 'age': 25, 'profession': 'Designer'},
    {'name': 'Peter', 'age': 40, 'profession': 'Programmer'},
    {'name': 'Maria', 'age': 28, 'profession': 'Manager'}
]

doc_ids = db.insert_multiple(users)
print(f"Documents added: {len(doc_ids)}")

Reading Data

Retrieve all documents:

all_users = db.all()
print(f"Total users: {len(all_users)}")

Retrieve a document by ID:

user = db.get(doc_id=1)
if user:
    print(f"Found user: {user['name']}")

Searching and Filtering

Basic search with Query:

from tinydb import Query
User = Query()

# Exact match
users = db.search(User.name == 'Ivan')

# Conditional match
adults = db.search(User.age >= 30)

# First matching document
user = db.get(User.name == 'Anna')

Complex search conditions:

# Logical AND
young_designers = db.search((User.age < 30) & (User.profession == 'Designer'))

# Logical OR
specialists = db.search((User.profession == 'Programmer') | (User.profession == 'Designer'))

# Logical NOT
not_managers = db.search(~(User.profession == 'Manager'))

Search within nested fields:

# Insert a document with nested structure
db.insert({
    'name': 'Alexey',
    'contact': {
        'email': 'alex@example.com',
        'phone': '+7-999-123-45-67'
    },
    'skills': ['Python', 'JavaScript', 'SQL']
})

# Query nested field
user = db.get(User.contact.email == 'alex@example.com')

Updating Data

Simple update:

# Update by condition
db.update({'age': 31}, User.name == 'Ivan')

# Update multiple fields
db.update({'age': 26, 'city': 'Saint Petersburg'}, User.name == 'Anna')

Update using functions:

# Increment age by 1
db.update(lambda doc: {'age': doc['age'] + 1}, User.age < 30)

# Add a new skill
def add_skill(doc):
    skills = doc.get('skills', [])
    if 'Docker' not in skills:
        skills.append('Docker')
    return {'skills': skills}

db.update(add_skill, User.name == 'Alexey')

Update by doc_id:

db.update({'status': 'active'}, doc_ids=[1, 2, 3])

Deleting Data

Delete by condition:

# Remove a specific user
db.remove(User.name == 'Peter')

# Remove by age
db.remove(User.age > 40)

Delete by doc_id:

db.remove(doc_ids=[1, 2, 3])

Clear an entire table:

db.truncate()

Working with Tables

Creating and Using Tables

TinyDB supports multiple tables within a single file:

from tinydb import TinyDB

db = TinyDB('app.json')

# Create tables
users = db.table('users')
products = db.table('products')
orders = db.table('orders')

# Table operations
users.insert({'name': 'Ivan', 'role': 'admin'})
products.insert({'name': 'Laptop', 'price': 50000})
orders.insert({'user_id': 1, 'product_id': 1, 'quantity': 1})

Retrieving Table Information

# List all tables
table_names = db.tables()
print(f"Available tables: {table_names}")

# Document count in a table
user_count = len(users)
print(f"Users: {user_count}")

# Check if a table exists
if 'users' in db.tables():
    print("The users table exists")

Dropping Tables

# Drop a single table
db.drop_table('old_data')

# Drop all tables
db.drop_tables()

Advanced Search Features

Value Comparison and Checks

from tinydb import Query

User = Query()

# Numeric comparisons
young_users = db.search(User.age < 25)
middle_aged = db.search(User.age.between(25, 35))

# String operations
name_contains = db.search(User.name.matches(r'.*an.*'))  # Regex
starts_with_a = db.search(User.name.startswith('A'))

# Field existence
has_email = db.search(User.email.exists())

# Non‑empty check
non_empty_skills = db.search(User.skills.exists() & (User.skills != []))

Working with Arrays

# Search by array elements
python_devs = db.search(User.skills.any(['Python']))
multi_skilled = db.search(User.skills.all(['Python', 'JavaScript']))

# Search by array length
experienced = db.search(User.skills.test(lambda x: len(x) > 3))

Custom Search Functions

# Custom search function
def is_senior_dev(doc):
    return (doc.get('age', 0) > 30 and 
            'Python' in doc.get('skills', []) and 
            doc.get('experience', 0) > 5)

senior_devs = db.search(User.test(is_senior_dev))

Configuration and Storage

Storage Types

from tinydb.storages import JSONStorage, MemoryStorage
from tinydb.middlewares import CachingMiddleware

# Standard JSON storage
db = TinyDB('db.json', storage=JSONStorage)

# In‑memory storage
db = TinyDB(storage=MemoryStorage)

# Cached storage
db = TinyDB('db.json', storage=CachingMiddleware(JSONStorage))

Performance Tuning

# Disable auto‑write for batch operations
db.storage.write_back = False

# Bulk insert
for i in range(1000):
    db.insert({'id': i, 'value': f'item_{i}'})

# Force write to disk
db.storage.flush()
db.storage.write_back = True

Creating a Custom Storage

from tinydb.storages import Storage
import json
import gzip

class CompressedJSONStorage(Storage):
    def __init__(self, path):
        self.path = path
    
    def read(self):
        try:
            with gzip.open(self.path, 'rt', encoding='utf-8') as f:
                return json.load(f)
        except FileNotFoundError:
            return {}
    
    def write(self, data):
        with gzip.open(self.path, 'wt', encoding='utf-8') as f:
            json.dump(data, f, ensure_ascii=False, indent=2)

# Usage
db = TinyDB('compressed.json.gz', storage=CompressedJSONStorage)

Performance Optimization

Indexing

# Create an index for frequently queried fields
from tinydb.operations import set

# TinyDB builds simple indexes automatically for repeated use
# For best performance, use numeric IDs
db.insert({'id': 1, 'name': 'Ivan', 'indexed_field': 'value'})

Batch Operations

# Efficient bulk insert
batch_data = [{'id': i, 'value': f'item_{i}'} for i in range(1000)]

# Disable auto‑write
with db.storage.write_back_disabled():
    db.insert_multiple(batch_data)

Query Optimization

# Use get() for single records
user = db.get(User.id == 123)  # Faster

# Use search() only for multiple results
users = db.search(User.id == 123)  # Slower for a single record

# Limit results
first_10_users = db.search(User.age > 18)[:10]

Integration with Web Frameworks

Flask Integration

from flask import Flask, request, jsonify
from tinydb import TinyDB, Query

app = Flask(__name__)
db = TinyDB('users.json')
User = Query()

@app.route('/users', methods=['GET'])
def get_users():
    users = db.all()
    return jsonify(users)

@app.route('/users', methods=['POST'])
def create_user():
    data = request.get_json()
    doc_id = db.insert(data)
    return jsonify({'id': doc_id, 'message': 'User created'})

@app.route('/users/<int:user_id>', methods=['GET'])
def get_user(user_id):
    user = db.get(doc_id=user_id)
    if user:
        return jsonify(user)
    return jsonify({'error': 'User not found'}), 404

@app.route('/users/<int:user_id>', methods=['PUT'])
def update_user(user_id):
    data = request.get_json()
    db.update(data, doc_ids=[user_id])
    return jsonify({'message': 'User updated'})

@app.route('/users/<int:user_id>', methods=['DELETE'])
def delete_user(user_id):
    db.remove(doc_ids=[user_id])
    return jsonify({'message': 'User deleted'})

if __name__ == '__main__':
    app.run(debug=True)

FastAPI Integration

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from tinydb import TinyDB, Query
from typing import List

app = FastAPI()
db = TinyDB('api_users.json')
User = Query()

class UserCreate(BaseModel):
    name: str
    age: int
    email: str

class UserResponse(BaseModel):
    id: int
    name: str
    age: int
    email: str

@app.post("/users/", response_model=UserResponse)
async def create_user(user: UserCreate):
    doc_id = db.insert(user.dict())
    return UserResponse(id=doc_id, **user.dict())

@app.get("/users/", response_model=List[UserResponse])
async def get_users():
    users = db.all()
    return [UserResponse(id=user.doc_id, **user) for user in users]

@app.get("/users/{user_id}", response_model=UserResponse)
async def get_user(user_id: int):
    user = db.get(doc_id=user_id)
    if not user:
        raise HTTPException(status_code=404, detail="User not found")
    return UserResponse(id=user.doc_id, **user)

Error Handling and Security

Error Handling

from tinydb.storages import JSONStorage
from tinydb import TinyDB
import json

try:
    db = TinyDB('db.json')
    user = db.get(User.id == 123)
except json.JSONDecodeError:
    print("JSON file read error")
    # Create backup and new DB
    import shutil
    shutil.move('db.json', 'db.json.backup')
    db = TinyDB('db.json')
except Exception as e:
    print(f"Unexpected error: {e}")

Data Validation

from jsonschema import validate, ValidationError

# User schema for validation
user_schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string", "minLength": 1},
        "age": {"type": "integer", "minimum": 0},
        "email": {"type": "string", "format": "email"}
    },
    "required": ["name", "age"]
}

def safe_insert(db, document):
    try:
        validate(document, user_schema)
        return db.insert(document)
    except ValidationError as e:
        print(f"Validation error: {e.message}")
        return None

# Usage
doc_id = safe_insert(db, {'name': 'Ivan', 'age': 30, 'email': 'ivan@example.com'})

File Locking

import fcntl
import json

class LockedJSONStorage:
    def __init__(self, path):
        self.path = path
    
    def read(self):
        try:
            with open(self.path, 'r', encoding='utf-8') as f:
                fcntl.flock(f.fileno(), fcntl.LOCK_SH)
                return json.load(f)
        except FileNotFoundError:
            return {}
    
    def write(self, data):
        with open(self.path, 'w', encoding='utf-8') as f:
            fcntl.flock(f.fileno(), fcntl.LOCK_EX)
            json.dump(data, f, ensure_ascii=False, indent=2)

# Use with locking
db = TinyDB('locked.json', storage=LockedJSONStorage)

Testing with TinyDB

Setting Up the Test Environment

import pytest
from tinydb import TinyDB
from tinydb.storages import MemoryStorage

@pytest.fixture
def test_db():
    """Create an in‑memory test database"""
    db = TinyDB(storage=MemoryStorage)
    
    # Pre‑populate with test data
    test_users = [
        {'name': 'Test User 1', 'age': 25, 'role': 'user'},
        {'name': 'Test User 2', 'age': 30, 'role': 'admin'},
        {'name': 'Test User 3', 'age': 22, 'role': 'user'}
    ]
    
    db.insert_multiple(test_users)
    yield db
    
    # Clean up after tests
    db.truncate()

def test_user_creation(test_db):
    """Test creating a user"""
    user_data = {'name': 'New User', 'age': 28, 'role': 'user'}
    doc_id = test_db.insert(user_data)
    
    assert doc_id is not None
    assert len(test_db.all()) == 4  # 3 preset + 1 new

def test_user_search(test_db):
    """Test searching users"""
    from tinydb import Query
    User = Query()
    
    admins = test_db.search(User.role == 'admin')
    assert len(admins) == 1
    assert admins[0]['name'] == 'Test User 2'

def test_user_update(test_db):
    """Test updating a user"""
    from tinydb import Query
    User = Query()
    
    test_db.update({'age': 26}, User.name == 'Test User 1')
    updated_user = test_db.get(User.name == 'Test User 1')
    
    assert updated_user['age'] == 26

Mocks and Stubs

from unittest.mock import patch, MagicMock
import pytest

def test_database_error_handling():
    """Test handling of database errors"""
    with patch('tinydb.TinyDB') as mock_db:
        mock_db.side_effect = Exception("Database connection failed")
        
        with pytest.raises(Exception) as exc_info:
            db = TinyDB('test.json')
        
        assert "Database connection failed" in str(exc_info.value)

Comprehensive Reference Table of Methods and Functions

Comparison with Other Solutions

Best Practices

When to Use TinyDB

TinyDB is ideal for:

• Prototyping and MVP development
• CLI utilities and scripts
• Local application configuration files
• Testing and development
• Educational projects
• Simple desktop applications
• Storing small data sets (up to ~10 000 records)

When Not to Use TinyDB

Avoid TinyDB when:

• High performance is critical
• Data volume exceeds ~10 000 records
• Complex JOIN‑like queries are needed
• ACID transactions are required
• Multiple users need concurrent access
• The application is mission‑critical

Performance Recommendations

# Good: Batch insertion
users = [{'name': f'User {i}', 'age': 20+i} for i in range(100)]
db.insert_multiple(users)

# Bad: Many single inserts
for i in range(100):
    db.insert({'name': f'User {i}', 'age': 20+i})

# Good: Use get() for single lookups
user = db.get(User.id == 123)

# Bad: Use search() for single lookups
users = db.search(User.id == 123)
user = users[0] if users else None

Data Structure Tips

# Good: Flat structure for fast queries
user = {
    'id': 123,
    'name': 'Ivan Petrov',
    'email': 'ivan@example.com',
    'age': 30,
    'city': 'Moscow',
    'role': 'admin'
}

# Acceptable: Nested structure for related data
user = {
    'id': 123,
    'name': 'Ivan Petrov',
    'contact': {
        'email': 'ivan@example.com',
        'phone': '+7-999-123-45-67'
    },
    'profile': {
        'age': 30,
        'city': 'Moscow',
        'bio': 'Python developer'
    }
}

Migrations and Versioning

Database Versioning System

from tinydb import TinyDB, Query

class DatabaseMigrator:
    def __init__(self, db_path):
        self.db = TinyDB(db_path)
        self.meta = self.db.table('_meta')
        
    def get_version(self):
        Meta = Query()
        version_doc = self.meta.get(Meta.key == 'version')
        return version_doc['value'] if version_doc else 0
    
    def set_version(self, version):
        Meta = Query()
        self.meta.upsert({'key': 'version', 'value': version}, Meta.key == 'version')
    
    def migrate_to_v2(self):
        """Add a 'status' field to all users"""
        users = self.db.table('users')
        User = Query()
        
        for user in users.all():
            if 'status' not in user:
                users.update({'status': 'active'}, doc_ids=[user.doc_id])
    
    def migrate_to_v3(self):
        """Convert 'age' to 'birth_year'"""
        users = self.db.table('users')
        from datetime import datetime
        
        current_year = datetime.now().year
        
        for user in users.all():
            if 'age' in user and 'birth_year' not in user:
                birth_year = current_year - user['age']
                users.update({'birth_year': birth_year}, doc_ids=[user.doc_id])
                # Remove old field
                new_user = {k: v for k, v in user.items() if k != 'age'}
                users.update(new_user, doc_ids=[user.doc_id])
    
    def migrate(self):
        current_version = self.get_version()
        
        if current_version < 2:
            print("Migrating to version 2...")
            self.migrate_to_v2()
            self.set_version(2)
        
        if current_version < 3:
            print("Migrating to version 3...")
            self.migrate_to_v3()
            self.set_version(3)
        
        print(f"Database upgraded to version {self.get_version()}")

# Usage
migrator = DatabaseMigrator('app.json')
migrator.migrate()

Backup and Recovery

Creating Backups

import shutil
import json
from datetime import datetime
from tinydb import TinyDB

class BackupManager:
    def __init__(self, db_path):
        self.db_path = db_path
        self.db = TinyDB(db_path)
    
    def create_backup(self, backup_dir='backups'):
        """Create a backup copy"""
        import os
        os.makedirs(backup_dir, exist_ok=True)
        
        timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
        backup_name = f"backup_{timestamp}.json"
        backup_path = os.path.join(backup_dir, backup_name)
        
        shutil.copy2(self.db_path, backup_path)
        return backup_path
    
    def restore_backup(self, backup_path):
        """Restore from a backup file"""
        self.db.close()
        shutil.copy2(backup_path, self.db_path)
        self.db = TinyDB(self.db_path)
    
    def export_to_json(self, export_path):
        """Export all data to a JSON file"""
        data = {
            'tables': {},
            'timestamp': datetime.now().isoformat()
        }
        
        for table_name in self.db.tables():
            table = self.db.table(table_name)
            data['tables'][table_name] = table.all()
        
        with open(export_path, 'w', encoding='utf-8') as f:
            json.dump(data, f, ensure_ascii=False, indent=2)
    
    def import_from_json(self, import_path):
        """Import data from a JSON file"""
        with open(import_path, 'r', encoding='utf-8') as f:
            data = json.load(f)
        
        for table_name, documents in data['tables'].items():
            table = self.db.table(table_name)
            table.truncate()
            table.insert_multiple(documents)

# Usage
backup_manager = BackupManager('app.json')
backup_path = backup_manager.create_backup()
print(f"Backup created: {backup_path}")

Frequently Asked Questions

Why is TinyDB slow with large data sets?

TinyDB loads the entire JSON file into memory for each operation. For large data volumes, consider using SQLite or PostgreSQL.

How can I ensure thread‑safety?

import threading
from tinydb import TinyDB

class ThreadSafeTinyDB:
    def __init__(self, *args, **kwargs):
        self.db = TinyDB(*args, **kwargs)
        self.lock = threading.Lock()
    
    def insert(self, *args, **kwargs):
        with self.lock:
            return self.db.insert(*args, **kwargs)
    
    def search(self, *args, **kwargs):
        with self.lock:
            return self.db.search(*args, **kwargs)
    
    def update(self, *args, **kwargs):
        with self.lock:
            return self.db.update(*args, **kwargs)
    
    def remove(self, *args, **kwargs):
        with self.lock:
            return self.db.remove(*args, **kwargs)

# Usage
db = ThreadSafeTinyDB('thread_safe.json')

How do I validate data before insertion?

from jsonschema import validate, ValidationError

def validate_user(user_data):
    schema = {
        "type": "object",
        "properties": {
            "name": {"type": "string", "minLength": 1},
            "age": {"type": "integer", "minimum": 0, "maximum": 150},
            "email": {"type": "string", "format": "email"}
        },
        "required": ["name", "age"]
    }
    
    try:
        validate(user_data, schema)
        return True
    except ValidationError as e:
        print(f"Validation error: {e.message}")
        return False

# Usage
user_data = {'name': 'Ivan', 'age': 30, 'email': 'ivan@example.com'}
if validate_user(user_data):
    db.insert(user_data)

How can I automate backups?

import schedule
import time
from datetime import datetime

def auto_backup():
    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    backup_path = f'backups/auto_backup_{timestamp}.json'
    
    try:
        backup_manager.create_backup()
        print(f"Automatic backup created: {backup_path}")
    except Exception as e:
        print(f"Backup error: {e}")

# Schedule every 6 hours
schedule.every(6).hours.do(auto_backup)

# Run scheduler in a separate thread
import threading

def backup_scheduler():
    while True:
        schedule.run_pending()
        time.sleep(60)

backup_thread = threading.Thread(target=backup_scheduler, daemon=True)
backup_thread.start()

How can I optimize query performance?

# Cache frequently used queries
from functools import lru_cache

@lru_cache(maxsize=128)
def get_user_by_email(email):
    return db.get(User.email == email)

# Use a dedicated index field for fast lookups
db.insert({
    'name': 'Ivan',
    'email': 'ivan@example.com',
    'email_hash': hash('ivan@example.com')  # Quick hash lookup
})

# Limit result sets
recent_users = db.search(User.created_at > yesterday)[:100]

How should I handle corrupted JSON files?

import json
import shutil
from tinydb import TinyDB

def safe_load_db(db_path):
    try:
        db = TinyDB(db_path)
        # Verify that the file can be read
        db.all()
        return db
    except json.JSONDecodeError:
        print("Database file is corrupted!")
        
        # Backup the corrupted file
        backup_path = f"{db_path}.corrupted.backup"
        shutil.copy2(db_path, backup_path)
        print(f"Corrupted file saved as: {backup_path}")
        
        # Create a fresh database
        db = TinyDB(db_path)
        return db

# Usage
db = safe_load_db('app.json')

Conclusion

TinyDB is an excellent choice for projects that prioritize simplicity and minimal dependencies. This lightweight NoSQL database is perfect for prototyping, CLI utilities, testing, and small‑scale applications.

Key advantages of TinyDB:

• Easy installation and usage
• Zero configuration
• Human‑readable JSON storage
• No external service dependencies
• Intuitive API
• Full integration with the Python ecosystem

When used appropriately, TinyDB can dramatically speed up development, especially during prototyping and MVP creation. However, it’s important to be aware of its limitations and avoid it for high‑load applications or massive data sets.

Thanks to its simplicity and efficiency, TinyDB remains a popular choice for developers who need to build functional solutions quickly without unnecessary complexity.