Forums

Let's talk DataTable!

This week, Jenny Donnelly and Satyam will join us to discuss the current state of DataTable in YUI 2 and 3, and where we are in the design/development phase for YUI 3's DataTable offering.

I'm simultaneously posting the YUI Blog announcement (http://www.yuiblog.com/blog/2010/08/16/ ... -aug-18th/), so I'll avoid repeating myself.

If there are any specifics that you want to address on the call, post them here to give us the heads up.

Hi Luke,

I would like to see freezing row/column feature in YUI 3. One hack I was trying to do as here

http://yuilibrary.com/projects/yui2/ticket/2032022

http://lravik.github.com/samples/dt_freeze_row.html

The recording worked!

The audio isn't stellar, but hey, baby steps. It's about 350M.

Despite my embarrassing disorganized banter, we'll continue to do this, and we'll post future links in the respective forum threads, and likely aggregate them somewhere centralized down the road a ways.

Satyam will be sharing his slides either here, or with me offline. In either case, there will be a link here.

There was a great deal of content, and as I listen to it again, I realize my notes are entirely inadequate. So really, you should just listen to it and watch the screen capture. Bear with the lengthy blocks of me filling the air with meaningless noise. Satyam and Jenny have good info, as well as Manuel Jasso who talked about some of the innovative work he and his colleagues are doing with DataTable at Wells Fargo.

Some other quick notes:
* DataTable is NOT a data grid, ala Excel, though we'd love to see a gallery contribution along these lines
* Use the DataTable's events (cellClickEvent, et al) instead of subscribing directly to cell content events
* Browsers hate rendering tables, so make them do it as little as possible (avoid creating large tables, especially when some or most of the table isn't visible, and try to modify table DOM in batches)

The link to the JSON.hpack work by Andrea Giammarchi here: http://wiki.github.com/WebReflection/json.hpack/

Options for DataTable in YUI 3 today:
1) use('yui2-datatable') which loads YUI 2.x DataTable into your YUI 3 env
2) use('gallery-datasource-wrapper') to use YUI 3's lighter DataSource with YUI 2's DataTable
3) use('gallery-simple-datatable') from apipkin and collaborate with him on its development (you'll find him in #yui on freenode)

Plan for YUI 3 DataTable:
Officially won't likely be in the library download until 3.4 (3.3 just got pulled into a rapid release cycle), BUT Jenny will be sharing early and often as she settles in on a base structure, a scrolling base structure, and some initial plugins. Keep your eye on the Gallery and YUIBlog for announcements about that.

Some wishlist items that I was able to capture:
* Decouple ordinal association between rows and records (bug filed)
* Add API or sugar for accessing rows by a data field identified as a primary key
* Configuration attribute to trigger a pre-render sort of the data on first fetch
* Embedding (at least) pagination controls in a tfoot
* Infinite/virtual scrolling of rows (lazy loading row data based on scroll position)
* Data filtering (by input/search or by row with filter dropdowns?)
* Some mechanism to batch UI updates to allow multiple data modifications without incurring reflows
* Row freeze, column freeze
* Simple use case = simple creation API. e.g. support instantiation without declaring both DS fields and DT columns
* Column resize that limits overall width to the table width (table width wins)
* Separate column resize plugin that expands/contracts the table width based on resize of the columns (column widths win)
* Break out RecordSet/Record into its separate modules (underway)
* Allow cell formatter capabilities for cell editors. Use the same default structure for bundled formatters/editors

Add to the list, +1 to things that interest you, debate the viability of one feature over another, and of course correct me if I didn't accurately capture (or I entirely missed) your wishlist items!

And PDF: http://yuiblog.com/sandbox/openhours/OH_Aug_18th.pdf

One of the messages in the chat window asked to have the Record and RecordSet classes as separate components. I also had a few items on my wishlist: to have a generic InputField class to deal with Cell Editors and also a Types table defining the parsers, formatters and editors used to fetch, show and edit a field. Since we got just to YUI3 DataTable Base by the end time, we didn't really had a chance to discuss all this. Let me put all those together and hopefully, invite some comments here.

It would be good to have a RecordSet class containing Record items. Such a class should have some direct means to be loaded from a DataSource via a DataSchema, there should be no need to have a DataTable mediating in between the local or remote source of data and the RecordSet where it will be stored internally.

A lot of the information that is currently set at the Columns Definition should be set at the RecordSet. Should the values be displayed at a DataTable or a Form, its default labels could be set at the RecordSet, to be overridden in each case if so desired.

Each field should be declared of a certain type. Types would simply be shortnames used as keys to both built-in and custom defined entries in a hash table containing several attributes for each field. Entries would include references to parsers, such as the ones used in YUI2 DataSource.responseSchema.fields array; formatters and editors such as those defined in the DataTable's Columns Definitions along with extra information such as it is usually provided to the constructor of the Cell Editor (validators, option values for dropdowns, radios and checkboxes, etc). It also needs a function to perform the complementary operation to parsing, that is, when the information needs to be sent back to the server, how to format the data back to a format the server would understand.

It would make sense to have this Types hash under the current DataTypes class: DataTypes.Types and it should be static much like YAHOO.widget.DataTable.Editors, YAHOO.widget.DataTable.Formatters and YAHOO.util.DataSource parsers are right now.

In order to help send information from the RecordSet back to the server, whole Record formatters should be provided, complementing those in DataSchema. The field value would first be formatted according to the field server-side formatter, for example, turned into a YYYY-MM-DD string for Dates and then the whole of the Record encoded. Plug-ins should be available to encode Records into JSON, XML, CSV or URL-Encoded and, for the first three, full RecordSets or filtered sub-sets of them should be assembled. The same 'key' properties used to fetch each field from the DataSouce/DataSchema on input would be used on its encoding.

RecordSets should be sortable. The sorted state should be part of the RecordSet properties, and might be used by widgets, such as DataTable to provide the visual clues to the user.

Each Record should have flags such as 'dirty' and 'selected'. They should not be states kept by the DataTable but by the Record object itself ('highlighted', in contrast, is purely UI related so the RecordSet has nothing to do with it). Perhaps a 'deleted' flag might be useful and widgets designed to display RecordSets should ignore or otherwise provide a visual clue for Records marked so. A 'deleted' flag might be useful if the deletion needs to be confirmed asynchronously with the server, or the developer wants to do it in batches and then, only after the operation gets confirmed, the records can actually be deleted.

A 'new' flag might be useful. A new but not dirty record is a record added but presumed empty since no actual data was actually entered. A new and dirty needs to be inserted into the DataBase table, a dirty but not new needs an update and a not dirty nor new has not been changed. Records marked deleted also become dirty.


A primaryKey property of the RecordSet could be used to build a hash to make it easy to access any individual Record. If no primaryKey is given, then no direct keyed access to the Records would be available, you would just have to loop through them all. A search by field value method could be provided, but if the field is not unique, it simply has to loop through it all and it returns an array instead of a single field as when the primaryKey field is used. This hash does not mean the Records should be stored in an object hash, since that would prevent them from being sorted.

Fields within Records might also have 'dirty' flags so that clean fields don't get encoded to be sent to the server. Fields marked as primaryKey always get encoded to be sent to the server, regardless whether they are dirty or not.

RecordSet definitions could be nested, much like is currently possible in DataTable with columns which would provide header nesting in a DataTable or enclosing the fields in a common <fieldset> in a form. Nesting fields would not have data fields but would have labels used for overarching headers or <legend> elements for <fieldsets>. Records need not reproduce this nesting as there should be no data for containing RecordSets, Records would always be flat.

RecordSet should provide methods to loop through all or some of the Records using filters to pick selected, dirty, deleted, new or any combination of those flags.

RecordSet data would be an array of Records or RecordSets thus allowing for grouping. I am not sure it would be wise to have Records and RecordSets mixed at the same level. It might be easier to have all Records at a fixed level of nesting, all the others being nesting RecordSets. Nesting RecordSets should allow for group-aggregates such as totals, counts or averages of the Records contained. Of course, this aggregates would all be plug-ins or extensions, not basic features, but the possibility of nesting has to be part of the base definition so search and enumeration features know how to deal with it. A mechanism should be provided to redo this grouping at any time.


The DataTable itself would be just one widget able to display such a RecordSet or a filtered subset. A Form widget could display individual Records and a Paginator might allow that Form to display successive Records. Paginator should be able to page through RecordSet groups, thus, we might have groups by initial or by ranges of, say, dates or values and Paginator should jump from one to the other, with the ability to dim out groups with no Records.

The DataTable would add to the Record object its own extra protected properties such as _trEl, which would point to the <tr> element displaying the data. If the RecordSet is sorted, the whole contents of the <tbody> containing them can be wiped out and the DataTable be redrawn by simply appending the <tr> elements pointed by _trEl as they come after the RecordSet is sorted. No further formatting of the values would be needed since the <tr> elements would have already gone through all needed formatting. If a field is changed, the whole <tr> element or the corresponding cell would then be deleted and reformatted, or somehow marked as pending so it will be re-formatted when needed.

Both Forms and DataTables should be able to accept input either directly by drawing an input element directly (the standard for forms) or by using a pop-up cell editor (the standard for DataTables). A DataTable might also be able to pop-up a form to do row-editing all at once. Both Form and DataTable definitions would allow the developer to select which fields to show and/or edit and if not stated otherwise, the settings of the RecordSet would be used for labels, editor and formatter. (validators are part of the editor definition)

I'm sure I have left something out, but I hope you guys will provide even more ideas.

Cheers.