huey

🦆 Huey

Huey is a static browser-based app that lets you explore and analyze data. Huey supports reading from multiple file formats, like .csv, .parquet, .json data files as well as .duckdb database files.

You can either load these files from your local workstation, or from a remote location via a URL; Or you can connect Huey to an external Data Lake or Quack server.

Try Huey now with some sample reports using the live demo at https://rpbouman.github.io/huey/src/index.html

(Note: this is a live demo that allows you to run Huey without even downloading it. Even though it’s available online, it’s still a static webapp: any data you load into it is safe, and stays on your local client.)

If you prefer to watch a video insted, here’s a few:

• 3rd DuckDB Meetup.
• DataZen talk.

Key features

• Supports .parquet, .csv, .json, .xlsx (MS Excel) both for analysis as well exporting results. Huey can also read DuckDB database files, or connect to a remote catalog.
• Comprehensive attributes tab to explore the structure of your dataset
• Intuitive query builder that supports projection, aggregation, filtering, and (sub)totals
• A pivot table to present analysis results
• Many different aggregate functions for reporting and data exploration
• Automatic breakdown of date/time columns to temporal hierarchy with derived attributes for year, month, quarter etc
• Extensive support for arrays and STRUCT data types to allow immediate ad-hoc analysis of complex, nested data (typical for JSON data)
• Export of result data and SQL queries to file or clipboard.
• Blazing fast, even for large files - courtesy of DuckDB
• Truly light-weight. Huey depends on DuckDb-WASM, and a Tabler Icons font, but nothing more. (Dependencies may be added in the future, but only when strictly necessary.)
• Accessible. Huey uses semantic HTML and aria-roles. Please let us know if you find Huey has accessibility issues!
• Run it your way!
  • Huey is a static webapp: you can simply download or check out the source tree, and open src/index.html in your browser (as file:// - no server required). But if you like, you can serve from any webserver like you would with any web page.
  • The latest stable release is available online as Live demo site. One click and you’re up and running! Even in this setup, any data you process with Huey remains local and private. There is no active server-side process.
  • Huey is also a progressive web app (PWA). This lets you install Huey on your device just as if it’s a native app. Running as PWA is described in more detail later in this readme.
• It’s free! Huey is released under the MIT license, just like DuckDB.

Limitations

• Huey is based on DuckDB-WASM (a DuckDB instance compiled to WebAssembly). DuckDB is awesome, and WASM is too! However, the WASM runtime imposes some limits which result in a poorer performance as compared to native DuckDB. (Note: this is not about DuckDB or DuckDB-WASM, the following limitations are mainly related to the WASM runtime) Most notably:
  • WASM in general is limited to a theoretical 4Gb memoery limit. In practice this may be closer to 3Gb. However, it’s complicated: DuckDB (and thus, DuckDB-WASM) uses advance streaming analytics, which often can process and aggregate datasets with volumes that exceed this limit.
  • WASM is confined to a single thread, eliminating some of DuckDB’s optimizations.
  • Some of the most interesting DuckDB capabilities, such as access to external remote databases such as MySQL, PostgreSQL, and the Unity Catalog rely on extensions that are not readily available to DuckDB-WASM That said, DuckDB-WASM is still incredibly fast when compared to any in-browser alternative. If you find that your use-case is stretching WASM’s limits or downright blocked, then you might try to use Huey as a client and connect to a DuckDB Server.
• Huey is subject to typical restrictions of a web application. In particular, URL datasources may not be directly accessible due to same-origin policy and/or missing CORS headers. If these issues are hampering your use case, the obvious step would be to server Huey over HTTP(S), and configure your webserver to pass the right headers. This is a common scenario for any web application; it’s just Huey is built in a way that also gets a lot done in completely static, offline scenarios. But that alone doesn’t mean you can’t or shouldn’t host Huey over HTTP(S)!
• Huey is best enjoyed on the desktop. Huey ought to work on a modern tablet. If you find it doesn’t, please submit an issue so this can be worked on. Huey does not work on your phone. If it does, then you just got lucky, but the experence will likely be frustrating because of the small screen size. There are currently no plans to make Huey work on a phone.

Getting started

For a super quick start, the following sections present sample reports using the live demo. All these examples use a URL to a publicly available dataset as datasource.

(For some of them you may need to wait a little. You’ll find that most of the time is spent waiting for the download while the analysis itself is pretty quick.)

CSV Examples

• Los Angeles International Airport: Number of Flight operations, by flight type and reporting period. This is from an open data set “Los Angeles International Airport - Flight Operations By Month” provided by the city of Los Angeles. The raw CSV data provides the number of flight operations, as well as a timestamp and various attributes that describe the flight, such as whether it is an arrival or departure; a charter or a scheduled flight, and whether it is a domestic or an international flight. The report demonstrates the following Huey features:
  • Reporting Period timestamp is correctly detected from the CSV data as TIMESTAMP type by the DuckDB CSV reader
  • Huey year and month derived attributes are applied on the row axis to produce a clear temporal breakdown
  • Flight type and Arrival/Departure are placed on the columns axis
  • Domestic/International is placed on the filters axis, showing all filter values in an expanded state. This lets you slice the data by checking/unchecking the checkbox for a specific filter value
  • The sum of the flight operations appears in the cells
  • (Sub)totals are included at the year, month and flight type level
• City of Chicago Energy Benchmarking: Use of electricity, gas and steam by district over years 2018 - 2022. This is from the Chicago Energy Benchmarking dataset provided by the Chicago Data Portal. The raw CSV data contains metrics for different kinds of energy and water consumption, along with the year and many columns identifying the location. The Huey report places the Community area on the rows axis (along with the first letter of the community area for quick alphabetic browsing), the year on the columns axis, and the sum of various energy consumption metrics in the cells. In addition, the report has an IN BETWEEN filter, allowing the data to be sliced in 3-year periods.
• Montgomery County of Maryland: Warehouse and Retail Sales.

JSON Examples

• Github Events: Actors against repo and time and event type. This sample uses “github’s public events endpoint”. The raw JSON data is an array of objects representing github events. The event object itself has scalar properties like id, type, and created_at timestamp, as well object-typed properties repo, actor and payload. The report demonstrates the following Huey features:
  • The DuckDB JSON reader correctly extracts the created_at string to a TIMESTAMP type.
  • Derived attributes year, month, and day as well as iso-time are applied on the column axis to produce a clean temporal breakdown
  • Event type also appears on the columns axis, showing CreateEvent, DeleteEvent, PushEvent and so on for each timestamp
  • The repo’s id and name properties are extracted from the repo-object nested inside the event object and placed on the rows Axis
  • In the cells, thelogin property extracted from the nested actor-object is aggregated using the list aggregator.

Parquet examples

• Train Services count across stations and service types The raw dataset contains trips from the Dutch train network. The huey report presents the stations on the rows and the train types on the columns, with the number of trips as cell values. It demonstrates the following Huey features:
  • on the columns, the “First letter” derived attribute is used to produce an alphabetic index for the train stations
  • on the columns the “No Case” derived attribute is used for a case-normalized presentation the train types. In the original dataset, the type column contains both the values Stoptrein and stoptrein; by using the “No Case” derived attribute, both are presented in a single column.

Running Locally

Want to run Huey locally? No problem! From the live demo, you can install the Huey progressive web app on your local device with just one click. Alternatively, you can download or check out the Huey source files and resources and run it as a local static webpage in your browser.

Running Huey on your Device as Progressive Web App (PWA)

1) Visit the live demo 2) Use your browser’s capabilities to install Huey on your local device. Typically this manifests itself as an installation icon in your browser’s URL bar:

After installing Huey as PWA, it appears just as if it is a local native app, and you should be able to find it using your operating system’s launch bar or start button:

PWA Offline

The Huey PWA caches itself automatically using your browser’s caching API. This requires a modest amount of storage, but the benefit is that this allows Huey to be used even when you’re completely disconnected from the internet.

PWA File Handler

The Huey PWA explicitly advertises the ability to open certain data files. Typically, the operating system picks this up and offer an “Open With” feature in the context menu of the file browser:

This way, you don’t even need to locate the app anymore - you just right click on a file you want to analyze, and choose “Open with Huey”.

Uninstalling the PWA

If you decide the PWA is not for you, that’s fine - you can always uninstall it using your operating system’s uninstall feature.

Running Huey from a folder on your device

1) Use git to check out the Huey github repository to a local folder, or download the repository as a .zip file and extract it to a folder. 2) Open index.html in your web browser. Note that although Huey runs locally, it depends on DuckDB-WASM and Tabler Icons, which are served by the jsdelivr.com CDN, so make sure you’re connected to the internet. Once these resources are downloaded, they are typically cached by your browser, often allowing you to run Huey even without an internet connection.

With either approach, the Huey files and resources are available in a folder of your choosing. Of course, if you checked out the repository you can use git pull to update too.

Registering and Analyzing Files with Huey

Huey uses DuckDb-WASM to read and analyze data files.

General browser security policies prevent web applications from autonomously accessing files on the local file system. Web application users need to explicitly select the files they want to analyze. Huey then registers them in DuckDB-WASM’s virtual file system so they become available for analysis.

Registering Files

To register one or more files, you can:

• Click the ‘Upload…’ button . The upload button is always available as the leftmost button on the toolbar at the top of the page. The upload action will pop up a file browser dialog that lets you browse and choose one or more files from your local filesystem. In the file browser dialog, navigate to the file or files that you want to explore, select them and then confirm the dialog by clicking the ‘Ok’ button.

  Note that by default, the File Browser dialog only lists files with a extension recognized by Huey. If your file doesn’t happen to have one of those known extensions, just choose “All files .”:

• If you installed Huey as PWA, and your files have an extension recognized by Huey, then you can typically also open it by right clicking the file and then choosing Open With from the context menu.

• Drag ‘n Drop one or multiple files unto the “Datasources” tab in the sidebar.

Upload Dialog

Either action will open the Upload dialog. The upload dialog will show a progress bar for each file that is being registered. Additional progress items may appear in case a duckdb extension needs to be installed and/or loaded.

After completion of the upload process, the upload dialog is updated to indicate the status of the uploads (or the extension installation, if applicable).

Items that encountered an error are indicated by red progressbars. In case of errors, the item is expanded to reveal any information that might help to remedy the issue.

Successful actions are indicated by green progressbars. Successfully loaded files are available in the Datasources tab, from where you can start exploring their contents by clicking the explore button . As a convenience, the explore button is also present in the upload dialog.

Datasource Tab

Successfully upload files are added to the Datasource tab. Each Datasource has a couple of buttons to work with the datasource:

• The Download Button lets you download the data. You will be prompted to choose the output format. You can read more about data export in the section about the Export dialog.
• The Datasource Settings button opens the Datasource Settings dialog.
• The Remove button removes the datasource from the tab.
• The Analyze Datasource button loads the datasource attributes into the Attributes tab so you can analyze its data using the Quey builder.

Huey will attempt to group files having similar column signature. The group appears as a separate top-level node in the Datasources tab, with its individual files indented below it. A file group has its own explore button, so that you can not only explore the individual files, but also the UNION of all Files in the group:

Files that cannot be grouped appear in a separate Miscellaneous Files group.

Opening DuckDb files

Apart from directly reading data files, Huey can also open existing DuckDB database files and access its tables and views.

The process for accessing duckdb files is exactly the same as for accessing data files. Huey assumes data files with a .duckdb extension are DuckDB database files.

If your DuckDB database file happens to have another extension - that’s totally fine! They just won’t appear automatically in the browser’s File Browser dialog, because by default, that only lists files with extensions that Huey knows about. You can always override that and select “All files (.)”.

Successfully loaded DuckDB database files appear in the DuckDb Folder, which appears at the top of the DataSources tab.

The schemas in the duckdb database file are presented as folders below the duckdb file entry, and any tables or views in the schema are presented below the schema folder. Each table or view has an explore button which you can click to explore the data.

Note: We ran into a limitation - when the duckdb file itself refers to external files, then it’s likely that Huey (or rather, DuckDB-WASM) won’t be able to find them. But native duckdb tables, as well as views based on duckdb base tables work marvelously and are quite a bit faster than querying bare data files.

Using Remote Datasets

Huey is not just for local files! You can also access remote data by registering a URL or [connecting to a remote Catalog](#remote-catalogs.

Register URLs

In addition to opening and registering local files, you can also register URLs. To register a URL, click the “Load data from URL” button on the toolbar load data from URL button . You will be prompted to enter the URL:

After confirming, the upload dialog appears just like when uploading local files.

Note that loading data from URL is subject to certain restrictions due to browser security policies. Typically the URL needs to be either in the same domain as from where Huey is served, or the remote server needs to pass CORS headers to overcome the same-origin policy.

In addition, some URLs might require authentication. If that is the case, you can use the Secrets Manager to create and maintain the secret.

Remote Catalogs

In Huey, you can use the Catalogs Manager to attach to remote databases. In this README, the Catalogs Manager is described in detail in its own section.

Exploring Datasources

The Datasources have an explore button . After clicking it, the left sidebar switches to the Attributes tab, which is then is populated with a list of the Attributes of the selected Datasource.

Attributes, Derived Attributes, and Aggregates

You can think of Attributes as a list of values (a column) that can be extracted from the Datasource and presented along the axes of the pivot table.

The pivot table has two axes for placing attribute values: 1) Attributes appearing on the horizontal axis are used to generate column headers. For this reason the horizontal axis is also known as the ‘columns’-axis. 2) attributes appearing on the vertical axis are used to generate row headers. For this reason the vertical axis is also known as the ‘rows’-axis.

The selection of attributes and their placement on the axis is represented by the Query Builder. The following screenshot may help to explain:

In the screenshot, the Attribute Sidebar is at the left side. The workarea is to the right of the Attribute Sidebar. The Query Builder is at the top right of the workarea. The pivot table is at the bottom right.

Searching Attributes

When there are a lot of attributes, it can be useful to use search to find them. To do that, use the Attribute Search feature: Simply type a part of the attribute name in the search input. After a brief timeout, the list of attributes will automatically show only those attributes that (partially) match the search string:

The search string is treated as a regular expression, and is matched in a case-insensitive manner, making it both easy and powerful. For extra convenience, the attribute searchstring also supports % as wildcard for any sequence of characters (just like the SQL LIKE operator).

Query Builder

The screenshot shows a simple query, with one attribute “hvfhs_license_num” placed on the columns axis of the Query Builder. Placing the attribute on the Columns axis causes its values to be shown as column headings of the pivot table.

Likewise, the attribute “dispatching_base_num” is placed on the Rows axis, and this causes its values to show as row headings in the pivot table.

Finally, the generic “count” aggregator is placed on the cells axis. This causes the value of the aggregate to be computed for each combination of values of the rows- and columns-headings. The aggregated value are placed in the cells at the intersection of the corresponding row and column.

By default, the cell headers appear on the Columns axis, below the last Column Axis item (if any). The cell headers can also by placed on the Rows axis, in which case they appear right to the values of the last row axis item:

(Note that for this particular example, which has only one aggregator on the cells-axis, its placement on either cells or rows doesn’t make much difference.)

Placing Attributes

Attributes can be placed by clicking one of the desired axis-placement buttons, which appear to the left of the attribute name.

Once the items are placed in the rows and column axes, you can move and flip the axes by clicking on the axis icon that appears right before the “Rows” and “Columns” axis header text. Clicking on the axis icon of the Cells axis will affect the placement of the cell headers on either of the Rows- and Columns- axes.

Items that are placed inside the Query builder have buttons to manipulate them: At the left and right side of the query items, there are buttons to move the item a single position to the left or right within the axis. Items on the rows and columns axes also have a button to move the item from one axis to the other. Items also have a button to remove it from the query.

Drag and Drop

Instead of using the buttons in the Attribute sidebar, you can also drag Attributes from the sidebar and drop them at the desired position in the Query Builder. Drag and Drop also works for items that are already placed inside the Query Builder.

Query Execution

After changing the Query, it must be executed so the pivot table may be updated. If the “Autorun query” checkbox on the toolbar is checked, this will happen automatically. If the “Autorun query” checkbox is not checked, then you can execute the query by clicking the “play” button that appears just in front of the checkbox label:

Derived Attributes

Right before the attribute item, there is a widget to expand the Attribute so its derived Attributes and Aggregates are revealed.

You can think of a derived attribute as an expression (formula) that calculates some aspect from a single value from the attribute upon which it is based. For example, from an attribute that represents timestamp values, we can extract only the date part, or only the time part, or even the individual parts like year, month, and so on. The values that are thus derived from the original attribute values can be thought of as a ‘virtual’ column.

Derived attributes may be placed on either the rows or the columns pivot table axis. Derived attributes may be used as filter too, and they can also be aggregated.

The attribute’s data type primarily determines which derived attributes are supported. Derived attributes tend to be grouped in folders around a similar use case or topic. Here’s a (non-exhaustive) list of derived attribute folders:

• date fields: Applies to attributes with temporal data types, like DATE and TIMESTAMP. Derived attributes in this category are mainly to present temporal data in a hierarchically organized breakdown, as well as to offer various formats and labels. Typical use cases are to compare data aggregated in the cells across years, month or day of the week.
• time fields: Similar to date fields, but for those data types that also carry a time part.
• string operations: Applies to attributes with a text type. This offers different collation variants of the text data to support case-insensitive filtering and/or sorting as well as actual case conversion.
• hashes various hash functions, which may be useful to compare large text fields
• array statistics: for calculating aggregates on the elements of array-typed attributes. Array statistics are described in more detail along with other types of aggregates.

Aggregates

Aggregates are expressions that calculate a single result on a collection of values. In statistics, this concept is also known as data reduction, as it takes many values as input to produce a single output value, which somehow represents the entire group of values. Typical examples of aggregate operations are counts, summation and averaging, but there are many more and more sophisticated aggregate operations.

To understand aggregates in Huey, it is useful to distinguish a few different ways to categorize aggregates:

• aggregation operation: This categorization refers to the method that is used to produce the output value based on the collection of input values. For example, a count simply returns the number of input values, while summation works by adding all the input values together and returning the total.

  The attributes tab of the sidebar has a generic count aggregator at the very top. All attributes also have a count and distinct count aggregator, which appears together with the derived attributes when you expand the attribute.

  Most attributes have a statistics folder which contains basic statistical descriptive operations like min (minimum), max (maximum), median and mode, as well as the special purpose entropy aggregator. In particular, numerical attributes have additional aggregators like sum (summation), avg (average), stdev (standard deviation), as well as more sophisticated ones like skewness and kurtosis.

  Most attributes also have a list aggregators folder. The aggregators in the list aggregators folder return a structure (like histogram) or a list as output value. List aggregators are not for general purpose, but can be useful for data exploration or text analysis.

  

• source or scope of the input values for the aggregation. In Huey, we can distinguish the following scopes: cell aggregates, axis aggregates, and array aggregates. Each is discussed in more detail below.

Cell-aggregates

Cell aggregates are items used to populate the cells of the query result. Cells refer to the intersections of the items appearing on the rows- and columns axes.

Cell aggregates take their inputs from all the rows in the underlying dataset that correspond with the values on the rows- and columns dataset. If we consider the entire pivot table as a SQL query, the cell aggregates would be plain aggregate functions in the SELECT-list, while the items on the row- and cells- axes would appear in the GROUP BY clause.

In Pivot tables, cell contents are always aggregate values. In the Query editor, cell-aggregates are created by simply placing an aggregate on the cells-axis.

Note that the cells axis only accepts aggregate items. This is intentional: the cells-axis is to define the content for the cells, and almost by definition, any given cell corresponds to a collection of rows, and thus requires an aggregator to produce a single value to populate the cell with.

In the context of OLAP and pivot tables, Cell-aggregates are the most common use case. But Huey also defines axis aggregates and array aggregates (discussed below).

Axis-aggregates

Axis aggregates are aggregate items that appear on any other axis than the cells axis.

Just like cell aggregates, axis aggregates also take values from the underlying rows as input values. But while cell aggregates are calculated on only the rows corresponding to the intersection of the rows- and columns- axes, axis aggregates are computed with respect to a particular partition of the items from the axis on which it is placed.

Functionally, axis aggregates can be used to calculate (sub)totals. In this regard they are somewhat similar to the totals feature, but without generating super aggregate rows.

If we consider the entire pivot table as a SQL query, axis aggregates would be window functions over a partition of items from the axis on which it is placed.

Axis aggregates are created by placing an aggregate item on any axis that is not the cells-axis. This can be done either by clicking the rows- or columns- button on an aggregate item in the Attributes tab. Alternatively, an aggregate item can be dragged from the Attributes tab to the axis. Finally, items on the cells axis can also be dragged and then dropped on a non-cells axis.

Once the item is placed, any non-aggregate items that appear right before the new axis aggregate item are used to define its partition.

Once the axis aggregate is created it can be freely positioned anywhere on its axis - this will not change the partition definition. If an item that is used in the partition definition of any axis-aggregates is moved or removed, then that item is removed from the partition definition of those axis-aggregates.

All aggregate operations that are available for cell-aggregates are also available as axis aggregates.

Array aggregates

While cell-aggregates and axis-aggregates take input values from the underlying rows, array aggregates apply to the elements of a single array-typed value. In the Attributes sidebar, array attributes can be found in the array statistics folder.

Array aggregates are technically derived attributes for array-typed attributes that happen to apply an aggregate function to the array elements. The repertoire of aggregate operations for aggregate elements is largely the same as for cells- and axis- aggregates.

Structured types, Arrays, and Maps

Attributes can have any kind of datatype, including composite or “nested” data types - that is, types whose values are not scalar, but which consist of multiple elements. Especially when exploring JSON files one is likely to encounter attributes having these types.

Structured Types

Values with a Structured type (STRUCT) are in the end just values, and can be projected on the query axis as such. Attributes of this type also have a “structure” folder that gives access to its members.

Members are also just attributes, and will have their own derived attributes and aggregates, in accordance with the member type. Of course, members that are themselves of a structured type have their own structure folder that gives access to its members.

Arrays

Arrays are also just values and can be treated as such.

Attributes of an array type have a set of “array operations” derived attributes:

• elements: unnests the array and projects the element value on a separate tuple. Just like with members of structured types, array elements are just like attributes and may have derived attributes and aggregates in accordance with their type.
• element indices: unnests the array, and projects the element index. If both elements and element indices appear together on the same axis, then they are unrolled at the same level, so that the indices and the element values refer to the same element.
• length: returns the length of the array.
• sort values: array value after sorting the elements
• unique values: (sorted) array value after removing the duplicate elements
• unique values length: length of the deduplicated array.

When the elements or element indices derived attributes are applied to multiple, independent attributes, then they are unrolled independently, in order of appearance on the axis.

Arrays also support a collection of “array statistics”:

Array statistics are special derived attributes that calculate an aggregate value over the array’s elements.

Maps

Maps are structured types that are somewhat similar to Arrays. While arrays have an ordered collection of elements with an associated integer index, Maps are an unordered collections of entries. Map entries are values (which can be of any type) which are uniquely identified by a key, which also may have any type.

Maps have a folder with map operations:

• entries gives access to the key- and value- derived attributes. These will unnest the map and project the key and/or value. Like array elements and element indices, the key- and value- derived attributes of the same map attribute are not independent but unrolled together.
• entry count: the number of entries in the map
• keyset: the (sorted) list of keys.

Filtering

The query editor supports a special Filters axis. Items placed on the filters axis represent conditions that are applied on the underlying dataset. Items can appear independently on the filter axis: they are not automatically visible in the query result, but items that appear on the filter axis may also (additionally) be placed on the rows or columns axis.

Immediately after placing a new item on the Filters axis, the Filter Dialog pops up right below the Filter axis item. The Filter dialog lets you choose values and operators to filter the query results.

Filter Types

The Filter type dropdown appears in the top of the Filter Dialog. Here you choose the operator that should be used to filter the data. The options are:

• Include: the values in the data must match any of the filter values exactly
• Exclude: rows from the data appear only when the value from the respective item does not match any of the filter values. (Negated include)
• Like: the values in the data must match the pattern of one or more filter values. The pattern is a simple SQL LIKE pattern which supports % (percent sign) as wildcard for zero or more arbitrary characters, and _ (underscore) as wildcard for a single arbitrary character.
• Not Like: the values in the data must not match the pattern of any of the filter values. (Negated Like)
• Between: the values of the data must be between the filter value-ranges.
• Not Between: the values of the data must not be between any of the filter value-ranges (Negated Between)

Array Filter Type

Array-valued attributes support filtering using the regular Include/Exclude and Between/Not Between filter types. In addition, 4 array-specific filter types are supported:

• Include Has Any: Include the rows if the array contains any of the selected filter values.
• Exclude Has Any: Exclude the rows if the array contains any of the selected filter values. (Negated Has Any)
• Include Has All: Include the rows only if the array contains all of the selected filter values.
• Exclude Has All: Exclude the rows only if the array contains all of the selected filter values. (Negated Has All)

Finding Filter Values

In the top of the Filter Dialog there is an input where you can type a value that will be used as a pattern for retrieving values from the respective item.

The pattern is used in a LIKE comparison to retrieve the items values. LIKE patterns support two wildcards:

• '%' (percent sign) is a wildcard for any character sequence, of any length;
• '_' (underscore) is a wildcard for any single character.

The input supports multiple patterns. To enter multiple patterns, separate them by a semi-colon. You can also paste delimited data (such as a range of cells from an Excel workbook) and paste them directly into the input.

The retrieved values are placed in the Picklist appearing below the input.

Right above the input, there appear two checkboxes:

• Apply all filters: when checked, the Filter Picklist will be populated with values from only rows that (in addition to the pattern in the input) also respect all other filter items (if present)
• Auto-wildcards: when checked, texts entered or pasted into the input are automatically pre- and postfixed with the '%' wildcard, effectively finding all values that contain the entered text.

Adding/Deleting Filter Values

Filter values can be applied to the filter item by finding them in the Filter Value Picklist and clicking them.

The Range filter types Between and Not Between require two values, and the Filter dialog will reveal two value lists when choosing these filter types. When picking a new value, a new range is added using that value as both lower and upper bound of the range:

The upper bound value is automatically selected, and picking another value from the picklist will overwrite the selected value:

You can always manually select a value in either value list to overwrite it with a new value.

Alternatively, values may be entered manually and added by clicking the button next to the input, or by hitting the Enter key on the keyboard.

Applying a value in this way while an already applied value is selected will overwrite the applied value with the new one. To enter multiple filter values at once, separate them with a semi-colon and then hit the button or the Enter key. When pasting multiple values from outside Huey, for example from Excel or a text editor, separators like newline and tab are automatically replaced with the semi-colon for ease of use.

Applied values may be removed by selecting them and then hitting the Clear Highlighted dialog button, or the Delete key on the keyboard. Hitting the Clear All button will remove all applied values.

Filter Dialog Buttons

• The Apply button will actually apply the chosen values to the filter and close the Filter Dialog.
• The Remove button will remove the filter item entirely from the Filter axis and close the Filter Dialog
• The Cancel button will close the Filter Dialog without changing the state of the filter item.

Hiding, Revealing and Toggling applied Filter Values

Once the filter values are applied, the Filter Item will show the number of values as well as a collapser/expander.

By default, the filter item is collapsed. Clicking the expander will reveal the filter items:

The filter values have a checkbox that allows you to enable or disable that value.

(Sub)totals

Items on the row or columns axis have a “totals” toggle-icon. When enabled, totals for that item will be displayed in a totals row or column.

Saving & Restoring your query

Saving Queries

Everytime you make a change to your query it will be encoded and appended to the URL as fragment (a.k.a. hash or anchor). You can bookmark the url and revisit it later, or you can copy the url from your browser’s address bar and share it.

Restoring Queries

You can load queries simply by navigating to the respective url (including the fragment). The fragment includes the query and a reference to the datasource - not the actual data itself.

When restoring the query, Huey checks if there is currently a datasource present that matches the referenced datasource’s name and column signature. If so, it will use it. If there is currently no datasource that matches the referenced one, Huey will prompt you so you can upload it.

If the referenced datasource does not exist, but there are other datasources that could satisfy the query (based on whether it includes all attributes mentioned in the query), then the prompt will offer those datasources too as alternatives:

If the datasource uses an URL as datasource, Huey will attempt to access it directly. If that succeeds, you won’t be prompted to confirm: Huey will simply load the remote datasource and restore the query.

Undo & Redo

Because the query state updates the page url, you can use the browser’s standard Back and Forward buttons to browse between different versions of your query.

Cloning the Huey Window

Hitting the Clone button on the toolbar will open a new instance of Huey in a new browser tab, while preserving the existing data sources as well as the current query.

Export

Huey provides an export dialog that lets you use the query and/or its results outside Huey. The export dialog can be opened by clicking the download button . This button is visible as soon as items are placed unto the query builder.

Note: if you want to export an entire datasource, then you can also go to the Datasource tab and click its Download button.

Title and Title template

In the heder of the export dialog, you can control the title of the query by using the title template. The title template can contain “fields” using ${<field-name>]} syntax, which will get dynamically replaced. The title field shows the resulting text after the replacement.

The fields are:

• ${datasource}: the name of the datasource used by the query.
• ${<axis>-items}: the list of captions of the items appearing on that axis, where <axis> is the name of the axis. So the actual fields are ${columns-items}, ${rows-items}, ${cells-items} and ${filter-items}.
• ${utc-timestamp}: The UTC timestamp,
• ${timestamp}: The local timestamp

Data structure

How the result data will be exported. The options are:

• Pivot: the shape of the result data follows the pivot table.
• Table: the data is tabular, creating rows containing the values from all query axes. This may be more convenient for further data analysis outside Huey.

Export Destination

This controls what to do with the exported data:

• File: the data will be downloaded to a file
• Clipboard: the data is copied to the clipboard so you can paste it another application.

Export Format Tabs

The export dialog lets you export query results in various file formats like csv, parquet, or JSON. Each file type has its own tab with settings specific to that format.

Note: the format-specific options configured in the Export dialog are in general also applied when downloading the contents of a datasource from the Datasource tab.

Delimited text

Use this to export delimited text, like CSV and TSV. Delimited text settings:

• Formatting: this controls value formatting
  • Date format: the DuckDB date format specifier to use for representing values of the DATE-datatype.
  • Timestamp format: the DuckDB date format specifier to use for representing values of the TIMESTAMP-datatype.
  • NULL-value string: A string to use to represent NULL-values.
• Delimiters: this controls how rows and fields are represented
  • Column delimiter: the field delimiter. There’s a list of suggestions available.
  • Quote character: the character used to quote values in case the value contains some meta-character.
  • Escape character: the character used to escape the quote character.

    Excel

    Use this to export the data to the Excel .xlsx-format.

    JSON

    Use this to export data to JSON or JSONL-format.

    Parquet

    Use this to export data to the .parquet-format.

    SQL

    Use this to export the SQL query that would produce the result dataset.

Note that this option does not export any data; only the SQL statement that would produce the query result.

Query

This option you export the state of the Querybuilder.

Note that this option does not export any data; only the structure of the Huey query is exported.

Settings

The settings dialog lets you control Huey’s behavior. You can open the settings dialog by clicking the “gear” icon, which is on the right in the top toolbar: . Settings are persisted in the browser’s local storage. Settings are organized in separate tabs:

Datasource Settings

This lets you control the behavior of datasources.

• The UNION loose typing checkbox controls how Huey detects whether datasources have the same column signature. When unchecked, exact data type matching is used; when checked, a more loose typing rule is applied.

Value formatting

This tab bundles all settings that controls the default appearance of values

• NULL-value label: a string that is used to symbolize NULL-values.
• Totals label: a string that is used to indicate the value is the total of the item indicated by the header.
• Use default locale: when checked, the Browser’s default locale(s) are used to format numbers. When unchecked, the Locale setting becomes editable.
• Locale: the custom locale to use. You need to uncheck the Use default locale checkbox to edit this setting
• Min. integer digits: the minimal number of digits to use to denote the integer part of numbers
• Min. fraction digits: the minimal number of fractional digits to denote
• Max. fraction digits: the maximum number of fractional digits to denote

Attributes

Settings for behavior of the Attributes tab in the sidebar.

• Auto-reveal attributes used in query?: Whether to automatcially expand attribute nodes to reveal all items used in the query also in the Attributes tab. This is useful when opening a query vi

Query

Controls the behavior of the query editor.

• Autorun: whether queries execute automatically after editing the query

Filter

Settings to control the Query’s filter behavior

• Search timeout (ms): the number of milliseconds to wait after user input before running a query to populate the filter picklist.
• Picklist pagesize: the number of rows to fetch per query to populate the filter picklist.

Pivot Table

Settings that control the appearance and behavior of the Pivot Table

• Max. cellwidth (ch): Columns grow according to the largest value, up to this value. The unit is ch - the number of characters.

Theme

• Themes dropdown: a dropdown showing the various themes/color schemes.

Secrets Manager

Huey includes a graphical user interface for DuckDB’s Secrets Manager. The Huey Secrets Manager is a dialog that lets you create, edit, and store DuckDB secrets for services like AWS S3, Google Cloud Storage, Azure Blob Storage, Hugging Face, and more.

Opening the Secrets Manager

You can open the Secrets Manager by clicking the Secrets Manager button from the right side of the main toolbar. This is what it looks like:

• On the left side of the Secrets Manager Dialog, there’s a list that presents the list of stored secrets. In the list, secrets are organized by type.
  In the screenshot above, 3 secrets are visible in the list. One is selected: it’s the secret called ‘my_secret’ of the ‘s3’ type.

• On the right side of the Secrets Manager Dialog, there are two tabs:

  • The Form tab presents all the secret’s details as a structured form.
  • The Code tab has a code editor that lets you view and edit the Secret using DuckDB’s CREATE SECRET-syntax.

Editing Secrets

Selecting a secret in the list loads it from storage and populates the form and code tabs with its details so you can edit it.

Secret Manager Form view

The Form is structured thus:

• Name and type fieldset: This is the secret’s ‘header’. It consists of the following items:
  • Name: a unique name for the secret. This corresponds to the secret_name element of the CREATE SECRET-syntax. Note that in Huey, the secret’s name is a required field.

  • Type: the secret’s type. This corresponds to the secret_type element of the CREATE SECRET-syntax.

    In the form, the secret type is just a text field, but there’s a list of suggestions for all secret types that corresponding to DuckDB’s core extensions.

  • Autoload: a checkbox to control whether this particular secret ought to be activated when Huey starts up. This is useful if you’re regularly accessing resources that require the secret.

    Secrets that are not auto-loaded can be manually activated when required.

• Key/Value Pairs fieldset: The secret’s details are specified as a list of Key/Value pairs. This fieldset lets you maintain a list of these key/value pairs that define the particulars of the secret.

  A key/value pair consists of the following controls:

  • Key field. This is the left-most textfield. The key field is mandatory. It’s just a text field, but it provides a list of suggestions based on the selected secret type.
  • Field Type. This is a drop down list that controls what kind of values can be entered for the field. The Field types are:
    • ☑: Checkbox, indicating the value is BOOLEAN and can have either a TRUE or a FALSE value.
    • txt: Plaintext field, indicating the value is a string value.
    • ***: Password field. This indicates a text value that is to be treated as a secret. Password fields use a password input type so their value is not immediately visible when editing. These values are encrypted when the secret is stored.
    • {…}: MAP-field. This indicates the value is itself a set of key/value pairs If a well-known value is entered in the Key field, then an appropriate default type is automatically selected. However, the dialog always lets you manually override the default.
  • Value field. This is the rightmost textfield. For the structured value-types Array and Map, this field does not exist. In these cases, the value is made up of key/value pairs that appear indented below the structured key type.

Secret Manager Code view

The Code tab lets you view and edit the secret as a DuckDB CREATE SECRET-statement:

The code editor is particularly useful if you already have the SQL for a secret and you want to quickly enter it into the Secrets Manager.

Note that the code editor shows the secret as plaintext.

Which key/value pairs are appropriate or allowed, depends primarily on the secet type. In addition, some key/value pairs depend on each other. Please refer to the DuckDB documentation of the corresponding extension to learn more about which key/value pairs you need to define a secret of a particular type.

Creating a new Secret

1) Open the Secrets Manager dialog and click the “Add Secret” button . This is on the left side of the Secrets Manager toolbar. Alternatively, you may also click the “Create a new secret”-hyperlink, which appears next to that toolbar button if you didn’t already select an existing secret.

You can now use either the Form-tab or the Code-tab to define the secret. 2) In the form tab, enter a name for your new secret. Each secret has its own, unique name.

Then, use the suggestions list to pick one of the well-known secret types. If Huey does not provide a suggestion for a secret type that you know should be valid, then you can always override the type and enter one manually.

If you want the secret to be automatically loaded when Huey starts, also check the Autoload checkbox.

In the key/value fieldset, a new blank entry is automatically created. Fill out at least one key/value entry is required.

• Fill out the key field. Huey provides suggestions for the key field for any known secret types. Selecting a suggestion, or entering a well-known type, automatically results in choosing a default field type. If you’re sure you need a particular key but the Huey suggestions list does not provide it, then you can always enter one manually.

• Choose the field type. If you chose a key from the suggestions list, or if you entered a key that is well-known and appropriate for the secret type, a default field type is automatically chosen for you. Again, you may override the field type if you’re sure you need to.

• Enter the field value. The choice of field type directly affects what values you can add in the value field. The field type directly affects the kind of data you can enter into the value field; for example, choosing the checkbox will turn the value input into a checkbox.

To work with the existing fields, use the action buttons to the left of the key field:

• You can add as many key/value pairs as you like, Just click the “Add key/value pair”-button that appears immediately before the key field to create a new one.
• To remove a key/value pair, click the “Remove key/value pair”-button .
• You can also move the key/value pairs around using the “Move key/value pair up” and “Move key/value pair down” -buttons. 3) You can switch to the Code tab to see the equivalent CREATE SECRET-statement. Alternatively, you could have pasted or entered a CREATE SECRET-statement, and then switch to the Form-tab, which would then be populated accordingly.

4) If the secret appears valid, the “Save Secret”-button will be available in the Secret Manager’s toolbar. Click it to store the secret.

Encryption of password fields

The main purpose of DuckDB secrets is to configure credentials to access datasources that require authentication. Naturally, credentials are sensitive data and should therefore be protected.

The Secrets Manager will automatically encrypt the value of all password-typed key/value pairs. This is implemented using AES-GCM-256 encryption using the browser’s built-in crypto library.

Encryption requires a password. You will be automatically prompted whenever a password is required:

The first time a password is required, the password itself is hashed, and the hashed value is stored so the store can check whether the entered password is correct. It is important to realize the password itself is never stored. This means that once the store is initialized with a password, you can only decrypt the documents in the store using that password. So, make sure you don’t lose it!

You can always change the password later on by clicking the “Change Password”-button . This is available on the right side of the Secrets Manager’s toolbar.

If you lose your password, there is no way to recover any of the encrypted fields. In this case you can delete all encrypted documents by clicking the “Reset Secrets Store”-button .

Activating, Deactivating and auto-loading secrets

In order to use a secret, it needs to be activated. Activating the secret simply means the equivalent CREATE SECRET-statement is executed so that DuckDB will apply it when required.

Secrets are automatically activated when saving a secret. Secrets that are marked for auto-load are also automatically activated on Huey startup. Activating a secret may result in a prompt for the password if the secret contains key/value pairs of the password-type.

If a secret is selected in the secrets list, the toolbar will show one of these buttons, depending on its activation status:

• Deactivated button , indicating the secret is currently not active.
• Activated button , indicating the secret is currently active. In addition, active secrets are marked up bold in the list. Hovering over the Activate/Deactive button reveals an action to change the state:
• if the secret is in the active state, clicking the corresponding toolbar button deactivas it
• if the secret is in the inactive state, clicking the corresponding toolbar button activates it

Catalogs Manager

Originally, DuckDB advertised itself primarily as an in-process (aka “embedded”) local-first data engine, rather than as a fully-fledged analytical relational database system. While DuckDB remains committed to its embedded database roots, there’s an ongoing trend to integrate with external relational datastores, in particular Data Lakehouses.

From the DuckDB perspective, such external datasources take the form of “attached” databases. The SQL syntax to achieve this is the ATTACH-statement. The ATTACH statment has a flexible, type-dependent options section that’s used to define the details of the attached database.

The “attachment” is an abstraction that encapsulates a lot of the details of the external datastore:

• In the most basic case, it’s simply a pointer to a static file for the local duckdb engine to access and manage. This is what happens when you attach a DuckDB or SQLIte database file.
• In other cases, the local DuckDB instance acts as engine to a central Data Lake (or rather Data Lakehouse); that is: a collection of essentially static, flat data files, but with an additional bookkeeping protocol on top to represent physical data files as logical table objects. Concrete examples of this are external Iceberg and Unity catalogs.
• In yet other cases, it’s more like a federated database, with the local duckdb instance acting no longer as data engine, but rather as a client of a remote database server. The emerging Quack wire protocol falls into this category, as do the MySQL and PostgreSQL extensions.
• A Miscellaneous bag of hybrid solutions that mix some elememnts of the aforementioned cases. Ducklake is an example that mostly resembles the aforementioned external Data Lake example, but where the protocol requires an external server for key elements of the bookkeeping process. Motherduck is an example that mostly resembles the federated database example, but where the extension smartly divides the load over the local embedded instance and the remote server.

Huey provides a Catalogs Manager, wich is a graphical user interface to create, maintain and store configuration data required to attach to remote datasources. The Catalogs Manager also integrates with the Secrets Manager, as attaching a remote catalog often requires authentication.

Opening the Catalogs Manager

You can open the Catalogs Manager by clicking the Catalogs Manager button from the left side of the main toolbar.

This is what it looks like:

• On the left side of the Catalogs Manager Dialog, there’s a list that presents the list of stored catalog definitions. In the list, catalogs are organized by type.
  In the screenshot above, 4 catalogs are visible in the list. One is selected: it’s the catalog called ‘nl_railway’ of the ‘ducklake’ type.

• On the right side of the Catalogs Manager Dialog, there are two tabs:

  • The Form tab presents catalog details as a structured form.
  • The Code tab has a code editor that lets you view and edit the Secret using DuckDB’s ATTACH-syntax.

Editing Catalog Definitions

Selecting a catalog in the list loads it from storage and populates the form and code tabs with its details so you can edit it.

Catalogs Manager Form view

The Form is structured thus:

• Name and type fieldset: This is the ‘header’. It consists of the following items:
  • URL: a URL that acts as address of the remote catalog. This corresponds to the database-path element of the ATTACH-statement. The URL typically has a schema and internal syntax that is dependent upon the catalog type. Alternatively, the URL may follow a URL scheme that identifies the object store of a cloud provider. It’s also possible that the specific URL syntax depends on the key/value pairs that make up the ATTACH-options. Please refer to the documentation of the extension that implements the catalog for detailed information about how to construct a URL for a particular kind of catalog.
  • Name: a unique name for the catalog. This corresponds to the database-alias element of the ATTACH-statement.

  • Type: the catalog type. This corresponds to the TYPE field that appears in the attach-options of the ATTACH-statement.

    In the DuckDB documentation for the ATTACH-statement, only sqlite is listed as an acceptable value. In practice, ATTACHing to remote catalogs is managed by specific extensions that introduce their own values for the TYPE field.

    In the form, the catalog type is just a text field, but there’s a list of suggestions for all catalog types that correspond to DuckDB’s core extensions.

  • Autoload: a checkbox to control whether this particular catalog ought to be attached when Huey starts up. This is useful if you’re regularly analyzing data from that catalog.

    Catalogs that are not auto-loaded can be manually attached when required.

• Key/Value Pairs fieldset: The catalog’s details are specified as a list of Key/Value pairs. This fieldset lets you maintain a list of these key/value pairs that define the options for attaching to this catalog.

  A key/value pair consists of the following controls:

  • Key field. This is the left-most textfield. The key field is mandatory. It’s just a text field, but it provides a list of suggestions based on the selected catalog type.
  • Field Type. This is a drop down list that controls what kind of values can be entered for the key/value pair. The Field types are:
    • ☑: Checkbox, indicating the value is BOOLEAN and can have either a TRUE or a FALSE value.
    • txt: Plaintext field, indicating the value is a string value.
    • ***: Password field. This indicates a text value that is to be treated as a secret. Password fields use a password input type so their value is not immediately visible when editing. These values are encrypted when the secret is stored.
    • {…}: MAP-field. This indicates the value is itself a set of key/value pairs If a well-known value is entered in the Key field, then an appropriate default type is automatically selected. However, the dialog always lets you manually override the default.
  • Value field. This is the rightmost textfield. For the structured value-types Array and Map, this field does not exist. In these cases, the value is made up of key/value pairs that appear indented below the structured key type.

Catalogs Manager Code view

The Code tab lets you view and edit the secret as a DuckDB ATTACH-statement:

The code editor is particularly useful if you already have the ATTACH-statement code and you want to quickly enter it into the Catalogs Manager.

Note that the code editor does not mask or redact any password fields.

Which key/value pairs are appropriate or allowed, depends primarily on the catalog type. In addition, some key/value pairs depend on each other.

Please refer to the DuckDB documentation of the corresponding extension to learn more about which key/value pairs are appropriate, and what values they should take.

Creating a new Catalog

1) Open the Catalogs Manager dialog and click the “Add Catalog” button . This is on the left side of the Catalogs Manager toolbar. Alternatively, you may also click the “Add new remote catalog”-hyperlink, which appears next to that toolbar button if you didn’t already select an existing catalog.

You can now use either the Form-tab or the Code-tab to define the options for attaching to the catalog. 2) In the form tab, enter a name for your new Catalog. Each Catalog has its own, unique name.

Then, use the suggestions list to pick one of the well-known catalog types. If Huey does not provide a suggestion for a catalog type that you know should be valid, then you can always override the type and enter one manually.

Then, enter the URL that serves as extrernal identifier for the remote catalog.

If you want the catalog to be automatically loaded when Huey starts, also check the Autoload checkbox.

In the key/value fieldset, a new blank entry is automatically created. You may or may not need to enter key/value pairs, depending upon the type of catalog. Please refer to the documentation of the extension that implements the catalog to discover which key/value pairs are appropriate for your use case.

To edit and create key/value pairs, do:

• Fill out the key field. Huey provides suggestions for the key field for any known secret types. Selecting a suggestion, or entering a well-known type, automatically results in choosing a default field type. If you’re sure you need a particular key but the Huey suggestions list does not provide it, then you can always enter one manually.

• Choose the field type. If you chose a key from the suggestions list, or if you entered a key that is well-known and appropriate for the secret type, a default field type is automatically chosen for you. Again, you may override the field type if you’re sure you need to.

• Enter the field value. The choice of field type directly affects what values you can add in the value field. The field type directly affects the kind of data you can enter into the value field; for example, choosing the checkbox will turn the value input into a checkbox.

To work with the existing fields, use the action buttons to the left of the key field:

• You can add as many key/value pairs as you like, Just click the “Add key/value pair”-button that appears immediately before the key field to create a new one.
• To remove a key/value pair, click the “Remove key/value pair”-button .
• You can also move the key/value pairs around using the “Move key/value pair up” and “Move key/value pair down” -buttons. 4) You can switch to the Code tab to see the equivalent ATTACH-statement. Alternatively, you can paste or enter an ATTACH-statement, and then switch to the Form-tab. The form will be automatically populated accordingly.

5) If the catalog appears valid, the “Save Catalog”-button will be available in the Catalogs Manager’s toolbar. Click it to store the secret.

Managing Catalog Authentication

Most Catalog extensions support a SECRET option in their ATTACH-options. If specified, this should typically refer to an existing DuckDB Secret created with the CREATE SECRET-statement.

In Huey, the Catalogs Manager integrates with the Secrets Manager by offering suggestions for the value field of a key/value pair with a SECRET key:

When attaching to the catalog, Huey scans the catalog definition for any occurrence of the SECRET. If it finds one, it will automatically activate the secret so DuckDB can use it to connect to the catalog.

As the secret is likely to contain an encrypted password-type key/value pair, this may in turn prompt you for the password of the secrets store. Keep in mind: this is the password that you used to initialize the secrets store - NOT a password specific to your catalog configuration. Huey needs the password to the secrets store to decrypt the password fields in the stored secret so it can then run the CREATE SECRET-statement.

Activating, Deactivating and auto-attaching Catalogs

In order to use a Catalog, it needs to be activated. Activating the Catalog simply means the equivalent ATTACH-statement will be executed so that DuckDB can refer to it by name. In Huey, activating the catalog will also add it as a datasource to the Datasources tab so you can browse its tables and views and select them for data analysis:

Catalogs are automcatically activated on save. Catalogs marked for auto-load are also automatically activated on Huey startup.

If a catalog is selected in the catalogs list, the toolbar will show one of these buttons, depending on its attachement status:

• Deactivated button , indicating the catalog is currently not attached.
• Activated button , indicating the catalog is currently attached. In addition, active catalogs are marked up bold in the list. Hovering over the Activate/Deactive button reveals an action to change the state:
• if the catalog is attached, clicking the button will detach it.
• if the catalog is detached, clicking the button attaches it.

Catalog Examples

Ducklabs exposes 2 catalogs that are used as demo datasets in the online duckdb shell.

nl_railway Ducklake Catalog

The nl_railway Ducklake Catalog contains a few tables with dutch railway service data. To connect to it from within Huey, follow the steps for creating a new Catalog and enter the following details:

• when using the Catalog Form, enter:
  • name: nl_railway
  • type: ducklake
  • url: https://blobs.duckdb.org/datalake/nl-railway.ducklake
• when using the Catalogs Manager’s code view, enter:

  ATTACH 'https://blobs.duckdb.org/datalake/nl-railway.ducklake'
  AS nl_railway (
    TYPE ducklake
  )    
  

  

• Save the catalog.

  If all goes well, the Catalog is now added to the Datasources tab:

  

TPCH Iceberg Catalog

The TPCH Iceberg Catalog is the TPCH benchmark dataset, served as Iceberg tables stored on S3.

To connect to it from within Huey, you first have to create a secret, so follow the steps for creating a new secret:

• when using the Secret Form, enter:
  • name: my_secret
  • type: s3 Then, enter the following key/value pairs:
  • KEY_ID, with type txt and value AKIATZA7ONMDJXWR4UVN
  • SECRET, with type *** (passsword) and value elm876/SqATe455MgkK6RB4fGORgNYep7/0GXajv
  • REGION, with type txt and value us-west-1
• when using the Secrets Manager Code view, enter this statement:

  CREATE OR REPLACE TEMPORARY SECRET my_secret (
    TYPE s3
  , KEY_ID 'AKIATZA7ONMDJXWR4UVN'
  , SECRET 'elm876/SqATe455MgkK6RB4fGORgNYep7/0GXajv'
  , REGION 'us-west-1'
  )
  

  

• Save your secret. You may be prompted to enter your password. If that’s the case then enter the password with which you initialized the store. If you never initialized the store, then the prompt will inform you that you need to initialize the store and provide detailed instructions to do so.

Once the secret is in place, you can follow the steps for creating a new Catalog and enter the following details:

• when using the Catalog Form, enter:
  • name: iceberg_dataset
  • type: iceberg
  • url: arn:aws:s3tables:us-east-1:259911478022:bucket/iceberg-on-the-browser You also need to create and fill out a few key/value pairs:
  • ENDPOINT_TYPE, with type txt and value s3_tables
  • SECRET, with type txt and value my_secret
• when using the Catalogs Manager’s code view, enter:

   ATTACH 'arn:aws:s3tables:us-east-1:259911478022:bucket/iceberg-on-the-browser'
   AS iceberg_dataset (
     TYPE iceberg 
   , ENDPOINT_TYPE 's3_tables'
   , SECRET 'my_secret'
   )
  

  

• Save the catalog.

If all goes well, the Catalog is now added to the Datasources tab:

Connecting to a Quack Server

Huey can also be used as a client to a Quack server. To get started, follow these steps:

• create a duckdb process that is to be server; for example, by running the DuckDB commeand line client.
• INSTALL the quack extension. LOAD it too:

  INSTALL quack;
  LOAD quack;
  

• CALL the quack_serve() macro, passing the url and secret token:

  CALL quack_serve(
    'quack:localhost'
  , token = 'super_secret'
  )
  

• Be sure to create some tables. For example, use the TPCH extension.

  INSTALL tpch;
  LOAD tpch;
  CALL dbgen( sf = 1 );
  

  Or attach the nl_railway ducklake:

  ATTACH 'https://blobs.duckdb.org/datalake/nl-railway.ducklake'
  AS nl_railway (
    TYPE ducklake
  )    
  

  After setting up the server,

• open Huey
• Open the Huey Secrets Manager and create a Secret for the same token that was passed to quack_serve():

  CREATE OR REPLACE
  SECRET quack_localhost (
     TYPE quack
  , TOKEN 'super_secret'
  )  
  

• Open the Hue Catalogs Manager to create a catalog entry for the Quack server:

  ATTACH 'quack:localhost:9494'
  AS try_quack (
    TYPE quack
  , SECRET 'quack_localhost'
  )  
  

• Activate the Quack server entry.

If all goes well, the Quack server will be added to the Datasource Panel. From there, you can expanmd the node to reveal the attached catalog, and the schemas within it. Finally, expanding the schema folders reveals the tables or views so you can analyze them like any other Huey Datasource.

Note that for Quack catalogs, Huey delegates execution to the remote Quack server. So in this particular scenario, Huey is really a client that allows the server to do all the heavy lifting. This can be useful in particular to overcome memory and trheading limitations specific to DuckDb-WASM.

Using Quack to connect to a DuckDB Data Hub

It was just mentioned that Quack can overcome some limitations of the WASM environment. The previous sections also explained how the Huey Catalogs Manager relies on specific extensions that allow access to remote, ‘foreign’ datasources through the ATTACH statement.

Unfortunately, not all DuckDB extensions are available to DuckDB-WASM. Not even all core extensions. For example, the extensions to ATTACH popular RDBMS products like MySQL and PostgreSQL are not available directly to DuckDB-WASM.

For such use-cases, this limitation may be overcome by running a central DuckDB instance that acts as a hub to the external datasources. In this scenario, a regular (non-WASM) DuckDB instance, for which all necessary extensions are available and installed and ATTACH-ed, would be setup as a Quack server.
This DuckDB instance acts as a hub (or proxy) to access such remote datasources.

Huey would simply create a Quack catalog entry for the hub and connect to it as client. The following diagram illustrates the approach:

(Please note that the image above is simply an illustration - you can set up your data hub in any way you like. If it is a DuckDB instance, it can be set up to attach any remote datasource for which an extensions is installed and configured. In theory, the hub need not even be a DuckDB instance, it could be kind of Quack implementation, not necessarily DuckDB’s)

Development, Releases, and contributions

Branches

The Huey repository has two important branches - dev and main. Active development is done on the dev branch.

(Pre-)Release

Once every while, typically every few weeks, ongoing developments are captured in a (pre-)release, which gets its own version number and a nickname. You can check out prior releases here: https://github.com/rpbouman/huey/releases

A new (pre-)release is triggered whenever a dependency is updated (currently, Huey has two dependencies - DuckDB-WASM and Tabler Icons). Other events that trigger a (pre-)release is when ongoing development of new features and bugfixes is deemed stable - or at least stable enough to focus on new developments.

When a couple of pre-releases have been found stable enough for production usage, a release is made and the work from the dev branch is merged into the main branch. The live demo at https://rpbouman.github.io/huey/src/index.html is a github page that is created right on top of the main branch. So, a merge to the main branch is what updates the version of the live demo.

If you just want to enjoy use of a stable version of Huey, you can either use the live demo, or check out or download the main branch. If you want to enjoy the latest developments, then you should use the dev branch.

Checking your Huey version

You can verify the current Huey version in the about dialog:

Note that this also gives info on the versions of Huey’s dependencies.

URL Query Parameters

Some Huey settings can be configured via URL query parameters. For typical usage it is not recommended to pass these parameters. The main purpose is for development, debugging and to solve compatibility issues.

• loglevel: Controls the DuckDB-WASM loglevel. Valid values are defined by the DuckDB-WASM LogLevel Enumeration. You can pass either the Enumeration keys (DEBUG, ERROR, INFO, NONE, WARNING) or their corresponding integer values.
• duckdb-wasm: DuckDB-WASM version as published on jsdelivr. See https://data.jsdelivr.com/v1/packages/npm/@duckdb/duckdb-wasm for a list of acceptable versions. The ability to control the DuckDB-WASM version is mostly to solve compatibility issues around DuckDB extensions, as some extensions required by the user may not yet be available for newer DuckDB-WASM versions.

Integrating and/or Embedding Huey

You can embed huey inside a frame on your own webpage and control the application by sending it commands using the postMessage()-method. Currently this experimental feature is under development and not documented in detail. Please check out src/PostMessageInterface/PostMessageTestbed.html for an example that illustrates this feature.

Contributions

Your contributions are welcome! You can contribute in many ways:

• filing an issue: If something isn’t working, or not working the way it’s supposed to, please file an issue! To ensure maximum effectivity, clearly describe the component that is having an issue, describe the observed behavior, describe the expected behavior, and describe how this issue may be reproduced. Please include your browserversion and operating system too. If your issue relies on a dataset, and you are at liberty to share that, then please include that too. Please label your issue as “bug”.
• Suggest a feature. You may also file issues to request or suggest features. If you’re looking for a feature you know from other tools which Huey doesn’t have, then its typically helpful if you name the product and its feature name.
• Become an internationalization contributor! Huey has simple but functional internationalization support. To get started with internationalization take a look at the Internationalization directory in the src folder. Inside, you’ll find a huey.i18n.labels.template.js file. To create your own translation, copy it, rename as a locale and store it in the i18n subdirectory. There are already some translations there. You don’t necessarily need to commit to doing a full new translation: many current translations are generated and can be substantially improved with a little human touch. To test a translation, use your browser’s language settings and make sure your locale is the preferred locale. If all goes well, Huey should immediately respond to the locale change. You can contribute your translation by sending a pull request.
• Fork the repo and send a pull request. If you filed an issue or feature request, or you found an existing issue and feature request, you can also consider picking it up and send a pull request. There is no guarantee that your PR would be accepted, but in general these would be welcome. Just make sure there is an issue filed already that can be referenced, so that it is clear what the PR is attempting to fix or what feature the PR aims to deliver.
• Become an advocate. If you like Huey, spread the word! Share it with your friends and colleagues, and help them get set up. If Huey was of some help to you in your daily work, consider writing a blog about it, or maybe present your use case at a meeting.

• Become a sponsor. Just click the “Sponsor” button at the top of the Huey github project page:

  

  Alternatively, you can sponsor Huey by making a donation.

• Consultancy: if need help using, installing or deploying Huey, you can always ask for help. If you require professional support, we can work something out too.
• Commission a feature. If you need custom development, or would like help building your own custom development, then contact me and we’ll negotiate the details.

Sponsorship & Commercial Support

If your organization relies on Huey, corporate sponsorship helps ensure long-term stability and maintenance. Huey values its partners and commits to reliability so your teams can depend on it.

Huey offers corporate procurement and invoiced billing via GitHub:

Tier	Monthly	Benefits
Bronze Backer	$100	Company identifier listed in project documentation.
Silver Partner	$250	Logo placement; prioritized issue review.
Gold Sponsor	$500	Prominent logo placement; direct roadmap consultation.

View Sponsorship Tiers

For custom vendor agreements or inquiries, contact us directly.

This site is open source. Improve this page.