Batch

Where and why do we use the Batch node?

The Batch node is used to group sequences of messages into batches.

Modes of operation

Group by Number of Messages

The Batch node can be configured to group messages into batches based on the number of messages in a batch.

For example, if you set the batch size to 5, then the first 5 messages will be grouped into a batch before being released. Then the next 5 messages will be grouped into a batch, and so on.

See the below example flow for what this looks like in practice.

Batching messages into 5 groups

Additionally, you can add overlap so that the last message is included in the next batch

Batching messages into 5 groups with overlap

Group by Time Interval

The Batch node can be configured to group messages into batches based on a time period.

For example, if you set the batch time to 2 seconds, then all messages received within 2 seconds will be grouped into a batch before being released. Then the next batch will be created for the next 2 seconds, and so on.

See the below example flow for what this looks like in practice.

Batching messages into 2 second groups

Concatenate sequences

The Batch node can be configured to concatenate message sequences into a single message.

For example, you could use this to duplicate the sequence of messages in a sequence or order them by topic.

See the below example flow for what this looks like in practice.

Duplicating a sequence of data

Batch filter and concatenate

Demo flows used in this article

[{"id":"52c7d8c93d68afb5","type":"tab","label":"Batch Node","disabled":false,"info":"","env":[]},{"id":"7aa502dddb9274d2","type":"group","z":"52c7d8c93d68afb5","style":{"stroke":"#b2b3bd","stroke-opacity":"1","fill":"#f2f3fb","fill-opacity":"0.5","label":true,"label-position":"nw","color":"#32333b"},"nodes":["e3a5c066.16333","9223c119.c5268","e4d07fa1.78c16","848f59e5.7528d8","cf1bbb5e.ba3e68","a1e311d5.4dca1","2776c823.77eba8","3c47b863c67393a2","93c3e8c551c02f95","572ede73fc15f038","5043a65dd26e130d"],"x":34,"y":439,"w":592,"h":382},{"id":"2701ed93a0fbc58d","type":"group","z":"52c7d8c93d68afb5","style":{"stroke":"#b2b3bd","stroke-opacity":"1","fill":"#f2f3fb","fill-opacity":"0.5","label":true,"label-position":"nw","color":"#32333b"},"nodes":["cf1fd796.197678","d1c8ddf0.99b4e","f727a5d3.ea1a28","f4d6dba4.7e8ab8","31b81865.611788","356c8654.2ad1aa"],"x":34,"y":899,"w":492,"h":242},{"id":"0b151c343c56c94c","type":"group","z":"52c7d8c93d68afb5","style":{"stroke":"#b2b3bd","stroke-opacity":"1","fill":"#f2f3fb","fill-opacity":"0.5","label":true,"label-position":"nw","color":"#32333b"},"nodes":["74853568.22b87c","8f4f683.99d1998","6c47ccb3.bb0184","49c2ac1.59a9354","311dd6b4.5aeb7a","e27c55b0.18e9c8","9e65f29a.69ca2","817acbfb.452af8"],"x":34,"y":59,"w":552,"h":302},{"id":"b857a13782d7c5d9","type":"group","z":"52c7d8c93d68afb5","style":{"stroke":"#b2b3bd","stroke-opacity":"1","fill":"#f2f3fb","fill-opacity":"0.5","label":true,"label-position":"nw","color":"#32333b"},"nodes":["53645699.a35c48","4cb873e6.f9996c","51089be3.4ecbf4","84c34533.6284a8","c7241026.18245","67d24449.028eec","5d909bfb.6faf44","7b289e4ad723a92a"],"x":614,"y":59,"w":592,"h":302},{"id":"3eda846bc0d54e59","type":"group","z":"52c7d8c93d68afb5","style":{"stroke":"#b2b3bd","stroke-opacity":"1","fill":"#f2f3fb","fill-opacity":"0.5","label":true,"label-position":"nw","color":"#32333b"},"nodes":["7f1ce95c.7ddbc8","3412e439.eda55c","e6f01877.16d558","c11e5c5f.876d6","e99c703b.f404","dbd6e8b8.cbf2b8","408f3032.eafc1","5137b2d0.f4838c","c571b56c.ae63b8","c548f2c.641141","4aa7d5ab1091553e","65261ee2e95176c2"],"x":614,"y":899,"w":632,"h":402},{"id":"5043a65dd26e130d","type":"junction","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","x":100,"y":580,"wires":[["93c3e8c551c02f95","3c47b863c67393a2"]]},{"id":"65261ee2e95176c2","type":"junction","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","x":980,"y":1040,"wires":[["3412e439.eda55c"]]},{"id":"e3a5c066.16333","type":"batch","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","name":"","mode":"interval","count":10,"overlap":0,"interval":"2","allowEmptySequence":false,"topics":[],"x":280,"y":660,"wires":[["9223c119.c5268"]]},{"id":"9223c119.c5268","type":"join","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","name":"","mode":"auto","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":150,"y":740,"wires":[["e4d07fa1.78c16"]]},{"id":"e4d07fa1.78c16","type":"debug","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":510,"y":740,"wires":[]},{"id":"848f59e5.7528d8","type":"comment","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","name":"↑ create message sequence received within 2s","info":"","x":270,"y":780,"wires":[]},{"id":"cf1bbb5e.ba3e68","type":"comment","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","name":"← join sequence to array","info":"","x":490,"y":660,"wires":[]},{"id":"a1e311d5.4dca1","type":"inject","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","name":"","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":480,"wires":[["5043a65dd26e130d"]]},{"id":"2776c823.77eba8","type":"delay","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","name":"Rate  limit 1msg/0.5s","pauseType":"rate","timeout":"1","timeoutUnits":"seconds","rate":"1","nbRateUnits":"0.5","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":460,"y":580,"wires":[["e3a5c066.16333"]]},{"id":"5842d64f.9fc608","type":"comment","z":"52c7d8c93d68afb5","name":"Example: Time-based Group Mode - Group messages received within 2s","info":"*Time-based Group mode* of batch node can be used to create new message sequences from incoming messages received within specified time range. \n","x":300,"y":420,"wires":[]},{"id":"cf1fd796.197678","type":"inject","z":"52c7d8c93d68afb5","g":"2701ed93a0fbc58d","name":"Array of 3 characters [\"a\", \"b\", \"c\"]","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"SEQ","payload":"[\"a\", \"b\", \"c\"]","payloadType":"json","x":210,"y":940,"wires":[["d1c8ddf0.99b4e"]]},{"id":"d1c8ddf0.99b4e","type":"split","z":"52c7d8c93d68afb5","g":"2701ed93a0fbc58d","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":430,"y":940,"wires":[["f727a5d3.ea1a28"]]},{"id":"f727a5d3.ea1a28","type":"batch","z":"52c7d8c93d68afb5","g":"2701ed93a0fbc58d","name":"","mode":"concat","count":10,"overlap":0,"interval":10,"allowEmptySequence":false,"topics":[{"topic":"SEQ"},{"topic":"SEQ"}],"x":190,"y":1020,"wires":[["f4d6dba4.7e8ab8"]]},{"id":"f4d6dba4.7e8ab8","type":"join","z":"52c7d8c93d68afb5","g":"2701ed93a0fbc58d","name":"","mode":"auto","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":190,"y":1100,"wires":[["31b81865.611788"]]},{"id":"31b81865.611788","type":"debug","z":"52c7d8c93d68afb5","g":"2701ed93a0fbc58d","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":410,"y":1100,"wires":[]},{"id":"356c8654.2ad1aa","type":"comment","z":"52c7d8c93d68afb5","g":"2701ed93a0fbc58d","name":"← Duplicate SEQ","info":"","x":400,"y":1020,"wires":[]},{"id":"c851c021.a9688","type":"comment","z":"52c7d8c93d68afb5","name":"Example: Concatenate Mode - Duplicate a sequence of data","info":"*Concatenate mode* of batch node can be used to combine input message sequences to create a new message sequence. Order of the sequences can be specified using message topic assigned to each message in a sequence.  Message sequence can be specified multiple times.\n","x":260,"y":880,"wires":[]},{"id":"3c47b863c67393a2","type":"change","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","name":"Reset","rules":[{"t":"set","p":"reset","pt":"msg","to":"true","tot":"bool"},{"t":"delete","p":"payload","pt":"msg"},{"t":"delete","p":"topic","pt":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":130,"y":660,"wires":[["e3a5c066.16333"]]},{"id":"3d208473.f31e1c","type":"comment","z":"52c7d8c93d68afb5","name":"Example: Number-based Group Mode - Group 5 consecutive messages","info":"","x":290,"y":40,"wires":[]},{"id":"74853568.22b87c","type":"batch","z":"52c7d8c93d68afb5","g":"0b151c343c56c94c","name":"","mode":"count","count":"5","overlap":0,"interval":"5","allowEmptySequence":false,"topics":[],"x":180,"y":220,"wires":[["8f4f683.99d1998"]]},{"id":"8f4f683.99d1998","type":"join","z":"52c7d8c93d68afb5","g":"0b151c343c56c94c","name":"","mode":"auto","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":"false","timeout":"","count":"","reduceRight":false,"x":170,"y":280,"wires":[["6c47ccb3.bb0184"]]},{"id":"6c47ccb3.bb0184","type":"debug","z":"52c7d8c93d68afb5","g":"0b151c343c56c94c","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":350,"y":280,"wires":[]},{"id":"49c2ac1.59a9354","type":"comment","z":"52c7d8c93d68afb5","g":"0b151c343c56c94c","name":"↑ create message sequence with 5 messages","info":"","x":290,"y":320,"wires":[]},{"id":"311dd6b4.5aeb7a","type":"comment","z":"52c7d8c93d68afb5","g":"0b151c343c56c94c","name":"← join sequence to array","info":"","x":390,"y":220,"wires":[]},{"id":"e27c55b0.18e9c8","type":"inject","z":"52c7d8c93d68afb5","g":"0b151c343c56c94c","name":"","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":100,"wires":[["9e65f29a.69ca2"]]},{"id":"9e65f29a.69ca2","type":"function","z":"52c7d8c93d68afb5","g":"0b151c343c56c94c","name":"send: 1..20","func":"for(var x = 1; x <= 20; x++) {\n    node.send({payload: x});\n}","outputs":1,"timeout":"","noerr":0,"initialize":"","finalize":"","libs":[],"x":170,"y":160,"wires":[["74853568.22b87c"]]},{"id":"817acbfb.452af8","type":"comment","z":"52c7d8c93d68afb5","g":"0b151c343c56c94c","name":"← send 20 msgs with numbers 1..20","info":"","x":420,"y":160,"wires":[]},{"id":"ecff527d.d64cb","type":"comment","z":"52c7d8c93d68afb5","name":"Example: Number-based Group Mode - 5 consecutive messages, overlap 1 msg","info":"","x":900,"y":40,"wires":[]},{"id":"53645699.a35c48","type":"batch","z":"52c7d8c93d68afb5","g":"b857a13782d7c5d9","name":"","mode":"count","count":"5","overlap":"1","interval":"5","allowEmptySequence":false,"topics":[],"x":760,"y":220,"wires":[["4cb873e6.f9996c"]]},{"id":"4cb873e6.f9996c","type":"join","z":"52c7d8c93d68afb5","g":"b857a13782d7c5d9","name":"","mode":"auto","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":"false","timeout":"","count":"","reduceRight":false,"x":750,"y":280,"wires":[["51089be3.4ecbf4"]]},{"id":"51089be3.4ecbf4","type":"debug","z":"52c7d8c93d68afb5","g":"b857a13782d7c5d9","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":930,"y":280,"wires":[]},{"id":"84c34533.6284a8","type":"comment","z":"52c7d8c93d68afb5","g":"b857a13782d7c5d9","name":"↑ create message sequence with 5 messages with overlap of 1 msg","info":"","x":940,"y":320,"wires":[]},{"id":"c7241026.18245","type":"comment","z":"52c7d8c93d68afb5","g":"b857a13782d7c5d9","name":"← join sequence to array","info":"","x":970,"y":220,"wires":[]},{"id":"67d24449.028eec","type":"inject","z":"52c7d8c93d68afb5","g":"b857a13782d7c5d9","name":"","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":720,"y":100,"wires":[["5d909bfb.6faf44"]]},{"id":"5d909bfb.6faf44","type":"function","z":"52c7d8c93d68afb5","g":"b857a13782d7c5d9","name":"send: 1..20","func":"for (var x = 1; x <= 20; x++) {\n    node.send({ payload: x });\n}","outputs":1,"timeout":"","noerr":0,"initialize":"","finalize":"","libs":[],"x":750,"y":160,"wires":[["53645699.a35c48"]]},{"id":"7b289e4ad723a92a","type":"comment","z":"52c7d8c93d68afb5","g":"b857a13782d7c5d9","name":"← send 20 msgs with numbers 1..20","info":"","x":1000,"y":160,"wires":[]},{"id":"93c3e8c551c02f95","type":"function","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","name":"send: 1..20","func":"for (var x = 1; x <= 20; x++) {\n    node.send({ payload: x });\n}","outputs":1,"timeout":"","noerr":0,"initialize":"","finalize":"","libs":[],"x":250,"y":580,"wires":[["2776c823.77eba8"]]},{"id":"572ede73fc15f038","type":"comment","z":"52c7d8c93d68afb5","g":"7aa502dddb9274d2","name":"↓ send 20 msgs with numbers 1..20","info":"","x":320,"y":540,"wires":[]},{"id":"7f1ce95c.7ddbc8","type":"join","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":"","mode":"auto","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":770,"y":1220,"wires":[["e6f01877.16d558"]]},{"id":"3412e439.eda55c","type":"batch","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":"","mode":"concat","count":10,"overlap":0,"interval":10,"allowEmptySequence":false,"topics":[{"topic":"NEG"},{"topic":"POS"}],"x":770,"y":1140,"wires":[["7f1ce95c.7ddbc8"]]},{"id":"e6f01877.16d558","type":"debug","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":1050,"y":1220,"wires":[]},{"id":"c11e5c5f.876d6","type":"change","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":"POS","rules":[{"t":"set","p":"topic","pt":"msg","to":"POS","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":887,"y":1020,"wires":[["65261ee2e95176c2"]]},{"id":"e99c703b.f404","type":"change","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":"NEG","rules":[{"t":"set","p":"topic","pt":"msg","to":"NEG","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":887,"y":1060,"wires":[["65261ee2e95176c2"]]},{"id":"dbd6e8b8.cbf2b8","type":"switch","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":">= 0? \\n < 0?","property":"payload","propertyType":"msg","rules":[{"t":"gt","v":"0","vt":"num"},{"t":"else"}],"checkall":"true","repair":true,"outputs":2,"x":750,"y":1040,"wires":[["c11e5c5f.876d6"],["e99c703b.f404"]]},{"id":"408f3032.eafc1","type":"split","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":930,"y":940,"wires":[["dbd6e8b8.cbf2b8"]]},{"id":"5137b2d0.f4838c","type":"inject","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":"","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"SEQ","payload":"[1,-6,-8,7,2,-3]","payloadType":"json","x":750,"y":940,"wires":[["408f3032.eafc1"]]},{"id":"c571b56c.ae63b8","type":"comment","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":"↑ Join the sequence of messages","info":"","x":850,"y":1260,"wires":[]},{"id":"c548f2c.641141","type":"comment","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":"← Order sequence of messages: NEG, POS","info":"","x":1050,"y":1140,"wires":[]},{"id":"16341de8ac839049","type":"comment","z":"52c7d8c93d68afb5","name":"Example: Concatenate Mode - Batch Filter & Concat","info":"*Concatenate mode* of batch node can be used to combine input message sequences to create a new message sequence. Order of the sequences can be specified using message topic assigned to each message in a sequence.  Message sequence can be specified multiple times.\n","x":810,"y":880,"wires":[]},{"id":"4aa7d5ab1091553e","type":"comment","z":"52c7d8c93d68afb5","g":"3eda846bc0d54e59","name":"Set topic \"POS\" \\n or \"NEG\"","info":"","x":1140,"y":1040,"wires":[]}]

Node Documentation

Creates sequences of messages based on various rules.

Details

There are three modes for creating message sequences:

Number of messages
groups messages into sequences of a given length. The overlap option specifies how many messages at the end of one sequence should be repeated at the start of the next sequence.
Time interval
groups messages that arrive within the specified interval. If no messages arrive within the interval, the node can optionally send on an empty message.
Concatenate Sequences
creates a message sequence by concatenating incoming sequences. Each message must have a msg.topic property and a msg.parts property identifying its sequence. The node is configured with a list of topic values to identify the order sequences are concatenated.

Storing messages

This node will buffer messages internally in order to work across sequences. The runtime setting nodeMessageBufferMaxLength can be used to limit how many messages nodes will buffer.

If a message is received with the msg.reset property set, the buffered messages are deleted and not sent.