Download Apache Log File Structure

Apache Camel: File. The File component provides access to file systems, allowing files to be processed by any other Camel Components or messages from other components to be saved to disk. URI formator. Where directory. Name represents the underlying file directory. You can append query options to the URI in the following format, ?option=value& option=value&amp.. Only directories. Camel supports only endpoints configured with a starting directory.

So the directory. Name must be a directory. If you want to consume a single file only, you can use the file. Name option, e. g. Name=thefilename. Also, the starting directory must not contain dynamic expressions with ${ } placeholders. Again use the file.

Automatic Reconfiguration. When configured from a File, Log4j has the ability to automatically detect changes to the configuration file and reconfigure itself. Log4j XML Configuration Primer. Basic example. Below is a basic xml configuration file for log4j that will get you started: <?xml version='1.0' encoding='UTF-8.

Apache Log File Structure Diagram

Name option to specify the dynamic part of the filename. Avoid reading files currently being written by another application. Beware the JDK File IO API is a bit limited in detecting whether another application is currently writing/copying a file. And the implementation can be different depending on OS platform as well. This could lead to that Camel thinks the file is not locked by another process and start consuming it. Therefore you have to do you own investigation what suites your environment. To help with this Camel provides different read.

File Component. The File component provides access to file systems, allowing files to be processed by any other Camel Components or messages from other components to.

Lock options and done. File. Name option that you can use. See also the section Consuming files from folders where others drop files directly. URI Options. Common. Name. Default Value. Descriptionauto. Createtrue. Automatically create missing directories in the file's pathname.

For the file consumer, that means creating the starting directory. For the file producer, it means the directory the files should be written to.

Size. 12. 8kb. Write buffer sized in bytes. Namenull. Use Expression such as File Language to dynamically set the filename. For consumers, it's used as a filename filter. For producers, it's used to evaluate the filename to write.

If an expression is set, it take precedence over the Camel. File. Name header.

Note: The header itself can also be an Expression). The expression options support both String and Expression types. If the expression is a String type, it is always evaluated using the File Language. If the expression is an Expression type, the specified Expression type is used - this allows you, for instance, to use OGNL expressions. For the consumer, you can use it to filter filenames, so you can for instance consume today's file using the File Language syntax: mydata- ${date: now: yyyy.

Apache Hadoop is an open-source software framework written in Java for distributed storage and distributed processing of very large data sets on computer clusters. Maven is a build automation tool used primarily for Java projects. The word maven means 'accumulator of knowledge' in Yiddish. [3] Maven addresses two aspects of. Triple store TDB. Persist your data using TDB, a native high performance triple store. TDB supports the full range of Jena APIs. Fuseki. Expose your triples as a. A complete message broker and full JMS 1.1 provider featuring clustering, distributed destinations and XA support with pluggable persistence (JDBC, BDB, JDBM) and.

MMdd}. txt. From Camel 2. Camel. Overrule. File. Name header which takes precedence over any existing Camel. File. Name header; the Camel. Overrule. File. Name is a header that is used only once, and makes it easier as this avoids to temporary store Camel.

File. Name and have to restore it afterwards. Flatten is used to flatten the file name path to strip any leading paths, so it's just the file name. This allows you to consume recursively into sub- directories, but when you eg write the files to another directory they will be written in a single directory. Setting this to true on the producer enforces that any file name recived in Camel.

File. Name header will be stripped for any leading paths. Camel 2. 9. 3: this option is used to specify the encoding of the file. You can use this on the consumer, to specify the encodings of the files, which allow Camel to know the charset it should load the file content in case the file content is being accessed. Likewise when writing a file, you can use this option to specify which charset to write the file as well.

See further below for a examples and more important details. And. Delete. On. Rename.

Failtrue. Camel 2. This option is not available for the FTP component. Using. Copyfalse. Camel 2. 1. 3. 1: Perform rename operations using a copy and delete strategy. This is primarily used in environments where the regular rename operation is unreliable (e. This option takes precedence over the copy. And. Delete. On. Rename.

Fail parameter that will automatically fall back to the copy and delete strategy, but only after additional delays. Consumer. Name. Default Value. Descriptioninitial. Delay. 10. 00. Milliseconds before polling the file/directory starts. Milliseconds before the next poll of the file/directory. Fixed. Delay. Controls if fixed delay or fixed rate is used.

See Scheduled. Executor. Service in JDK for details. In Camel 2. 7. x or older the default value is false. From Camel 2. 8 onwards the default value is true. Logging. Level. TRACECamel 2. The consumer logs a start/complete log line when it polls. This option allows you to configure the logging level for that.

If a directory, will look for files in all the sub- directories as well. If true, the file will be deleted after it is processed successfully. If true, the file is not moved or deleted in any way. This option is good for readonly data, or for ETL type requirements.

If noop=true, Camel will set idempotent=true as well, to avoid consuming the same files over and over again. Movenull. Expression (such as File Language) used to dynamically set the filename when moving it before processing.

For example to move in- progress files into the order directory set this value to order. Expression (such as File Language) used to dynamically set the filename when moving it after processing. To move files into a . Failednull. Expression (such as File Language) used to dynamically set a different target directory when moving files in case of processing (configured via move defined above) failed. For example, to move files into a . Note: When moving the files to the “fail” location Camel will handle the error and will not pick up the file again. Is used to include files, if filename matches the regex pattern (matching is case in- sensitive from Camel 2.

Is used to exclude files, if filename matches the regex pattern (matching is case in- sensitive from Camel 2. Includenull. Camel 2.

Ant style filter inclusion, for example ant. Include=*/. txt. Multiple inclusions may be specified in comma- delimited format. See below for more details about ant path filters. Excludenull. Camel 2. Ant style filter exclusion.

If both ant. Include and ant. Exclude are used, ant.

Exclude takes precedence over ant. Include. Multiple exclusions may be specified in comma- delimited format. See below for more details about ant path filters. Filter. Case. Sensitivetrue. Camel 2. 1. 1: Ant style filter which is case sensitive or not.

Option to use the Idempotent Consumer EIP pattern to let Camel skip already processed files. Will by default use a memory based LRUCache that holds 1. If noop=true then idempotent will be enabled as well to avoid consuming the same files over and over again. Key. Expression. Camel 2. To use a custom idempotent key. By default the absolute path of the file is used.

You can use the File Language, for example to use the file name and file size, you can do. Repositorynull. A pluggable repository org. Idempotent. Repository which by default use Memory. Message. Id. Repository if none is specified and idempotent is true. Progress. Repositorymemory. A pluggable in- progress repository org.

Idempotent. Repository. The in- progress repository is used to account the current in progress files being consumed. By default a memory based repository is used. Pluggable filter as a org. Generic. File. Filter class.

Will skip files if filter returns false in its accept() method. More details in section below. Camel 2. 1. 6: To shuffle the list of files (sort in random order)sorternull.

Pluggable sorter as a java. Comparator< org.

Generic. File> class. Bynull. Built- in sort using the File Language. Supports nested sorts, so you can have a sort by file name and as a 2nd group sort by modified date. See sorting section below for details. Locknone. Used by consumer, to only poll the files if it has exclusive read- lock on the file (i. Camel will wait until the file lock is granted. This option provides the build in strategies:  none is for no read locks at all.

File Camel creates a marker file (file. Name. camel. Lock) and then holds a lock on it. This option is not available for the FTP component. Will at least use 1 sec. JDK IO API cannot always determine whether a file is currently being used by another process.

The option read. Lock. Check. Interval can be used to set the check frequency. This option is only avail for the FTP component from Camel 2. Notice that from Camel 2.

FTP option fast. Exists. Check can be enabled to speedup this read. Lock strategy, if the FTP server support the LIST operation with a full file name (some servers may not). Lock is for using java. File. Lock. This option is not avail for the FTP component.

This approach should be avoided when accessing a remote file system via a mount/share unless that file system supports distributed file locks. Camel 2. 1. 6 (only file component) is for using a idempotent. Repository as the read- lock.

This allows to use read locks that supports clustering if the idempotent repository implementation supports that. Warning: Most of the read lock strategies are not suitable for use in clustered mode. That is, you cannot have multiple consumers attempting to read the same file in the same directory. In this case, the read locks will not function reliably.

The idempotent read lock supports clustered reliably if you use a cluster aware idempotent repository implementation such as from Hazelcast Component or Infinispan. Lock. Timeout. 10.

Optional timeout in millis for the read- lock, if supported by the read- lock. If the read- lock could not be granted and the timeout triggered, then Camel will skip the file. At next poll Camel, will try the file again, and this time maybe the read- lock could be granted. Use a value of 0 or lower to indicate forever. In Camel 2. 0 the default value is 0. Starting with Camel 2. Currently file. Lock, changed and rename support the timeout.

Notice: For FTP the default read. Lock. Timeout value is 2.

The read. Lock. Timeout value must be higher than read. Lock. Check. Interval, but a rule of thumb is to have a timeout that is at least 2 or more times higher than the read. Lock. Check. Interval.

This is needed to ensure that amble time is allowed for the read lock process to try to grab the lock before the timeout was hit. Lock. Check. Interval. Camel 2. 6: Interval in millis for the read- lock, if supported by the read lock. This interval is used for sleeping between attempts to acquire the read lock. For example when using the changed read lock, you can set a higher interval period to cater for slow writes.

The default of 1 sec. For FTP the default read. Lock. Check. Interval is 5. The read. Lock. Timeout value must be higher than read. Lock. Check. Interval, but a rule of thumb is to have a timeout that is at least 2 or more times higher than the read. Lock. Check. Interval.

This is needed to ensure that amble time is allowed for the read lock process to try to grab the lock before the timeout was hit. Lock. Min. Length.

Camel 2. 1. 0. 1: This option applied only for read. Lock=changed. This option allows you to configure a minimum file length. By default Camel expects the file to contain data, and thus the default value is 1. You can set this option to zero, to allow consuming zero- length files.

Lock. Min. Age. 0Camel 2. This option applied only for read. Lock=change. This options allows to specify a minimum age the file must be before attempting to acquire the read lock. For example use read. Lock. Min. Age=3.

This can speedup the changed read lock as it will only attempt to acquire files which are at least that given age. Notice for FTP users then file timestamps reported from FTP servers often are only in minutes precision, so the min age check would often requires to be reported in minutes, eg 6.

Notice Camel support specifying this as 6. Lock. Logging. Level. WARNCamel 2. 1. 2: Logging level used when a read lock could not be acquired.

By default a WARN is logged. You can change this level, for example to OFF to not have any logging. This option is only applicable for read. Lock of types: changed, file. Lock, rename. read.

Lock. Marker. Filetrue. Camel 2. 1. 4: Whether to use marker file with the changed, rename, or exclusive read lock types. By default a marker file is used as well to guard against other processes picking up the same files. This behavior can be turned off by setting this option to false. For example if you do not want to write marker files to the file systems by the Camel application. Lock. Remove. On. Rollbacktrue. Camel 2.

This option applied only for read. Lock=idempotent. This option allows to specify whether to remove the file name entry from the idempotent repository when processing the file failed and a rollback happens. If this option is false, then the file name entry is confirmed (as if the file did a commit).

Lock. Remove. On. Commitfalse. Camel 2. This option applied only for read. Lock=idempotent. This option allows to specify whether to remove the file name entry from the idempotent repository when processing the file succeeded and a commit happens. By default the file is not removed which ensures that any race- condition do not occur so another active node may attempt to grab the file.

Instead the idempotent repository may support eviction strategies that you can configure to evict the file name entry after X minutes - this ensures no problems with race conditions. Lock. Delete. Orphan. Lock. Filestrue. Camel 2. Whether or not read lock with marker files should upon startup delete any orphan read lock files, which may have been left on the file system, if Camel was not properly shutdown (such as a JVM crash). If turning this option to false then any orphaned lock file will cause Camel to not attempt to pickup that file, this could also be due another node is concurrently reading files from the same shared directory. Must. Existfalse.

Camel 2. 5: Similar to starting. Directory. Must. Exist but this applies during polling recursive sub directories. File. Namenull. Camel 2. If provided, Camel will only consume files if a done file exists. This option configures what file name to use. Either you can specify a fixed name.

Or you can use dynamic placeholders. The done file is always expected in the same folder as the original file. See using done file and writing done file sections for examples. Read. Lock. Strategynull.

Pluggable read- lock as a org. Generic. File. Exclusive. Read. Lock. Strategy implementation. Messages. Per. Poll.

An integer to define a maximum messages to gather per poll. By default no maximum is set. Can be used to set a limit of e. Set a value of 0 or negative to disabled it.

See more details at Batch Consumer. Notice: If this option is in use then the File and FTP components will limit before any sorting. For example if you have 1. Messages. Per. Poll=5. You can use the eager. Max. Messages. Per. Poll option and set this to false to allow to scan all files first and then sort afterwards.

Max. Messages. Per. Polltrue. Camel 2. Allows for controlling whether the limit from max. Messages. Per. Poll is eager or not. If eager then the limit is during the scanning of files. Where as false would scan all files, and then perform sorting.

Setting this option to false allows for sorting all files first, and then limit the poll. Mind that this requires a higher memory usage as all file details are in memory to perform the sorting. Depth. 0Camel 2. 8: The minimum depth to start processing when recursively processing a directory. Using min. Depth=1 means the base directory.

Using min. Depth=2 means the first sub directory. This option is supported by FTP consumer from Camel 2.

Depth. Integer. MAX_VALUECamel 2. The maximum depth to traverse when recursively processing a directory. This option is supported by FTP consumer from Camel 2.

Strategynull. A pluggable org. Generic. File. Process. Strategy allowing you to implement your own read. Lock option or similar.

Can also be used when special conditions must be met before a file can be consumed, such as a special ready file exists. If this option is set then the read. Lock option does not apply. Directory. Must. Existfalse. Camel 2. 5: Whether the starting directory must exist. Mind that the auto.

Create option is default enabled, which means the starting directory is normally auto created if it doesn't exist. You can disable auto. Create and enable this to ensure the starting directory must exist. Will thrown an exception if the directory doesn't exist.

Strategynull. A pluggable org. Polling. Consumer. Poll. Strategy allowing you to provide your custom implementation to control error handling usually occurred during the poll operation before an Exchange have been created and being routed in Camel.

In other words the error occurred while the polling was gathering information, for instance access to a file network failed so Camel cannot access it to scan for files. The default implementation will log the caused exception at WARN level and ignore it.

Empty. Message. When. Idlefalse. Camel 2.

If the polling consumer did not poll any files, you can enable this option to send an empty message (no body) instead. Error. Handlerfalse. Camel 2. 1. 0: Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while trying to pickup files, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org. Exception. Handler to deal with exceptions, that by default will be logged at WARN/ERROR level and ignored. See further below on this page fore more details, at section How to use the Camel error handler to deal with exceptions triggered outside the routing engine.

Executor. Servicenull. Camel 2. 1. 0: Allows for configuring a custom/shared thread pool to use for the consumer.

By default each consumer has its own single threaded thread pool. This option allows you to share a thread pool among multiple file consumers. Camel 2. 1. 2: To use a custom scheduler to trigger the consumer to run. See more details at Polling Consumer, for example there is a Quartz. Spring based scheduler that supports CRON expressions.

Multiplier. 0Camel 2. To let the scheduled polling consumer backoff if there has been a number of subsequent idles/errors in a row. The multiplier is then the number of polls that will be skipped before the next actual attempt is happening again. When this option is in use then backoff. Idle. Threshold and/or backoff. Error. Threshold must also be configured. See more details at Polling Consumer.

Idle. Threshold. 0Camel 2. The number of subsequent idle polls that should happen before the backoff. Multipler should kick- in. Error. Threshold. Camel 2. 1. 2: The number of subsequent error polls (failed due some error) that should happen before the backoff. Multipler should kick- in.

Completion. Exception. Handler Camel 2. 1. To use a custom org. Exception. Handler to handle any thrown exceptions that happens during the file on completion process where the consumer does either a commit or rollback. The default implementation will log any exception at WARN level and ignore. Default behavior for file consumer.

By default the file is not locked for the duration of the processing. After the route has completed, files are moved into the . The File Consumer will always skip any file whose name starts with a dot, such as ., .

Only files (not directories) are matched for valid filename, if options such as: include or exclude are used. Producer. Name. Default Value. Descriptionfile. Exist. Override. What to do if a file already exists with the same name. The following values can be specified: Override, Append, Fail, Ignore, Move, and Try.

Rename (Camel 2. 1. Override, which is the default, replaces the existing file.

Append adds content to the existing file. Fail throws a Generic. File. Operation. Exception, indicating that there is already an existing file. Ignore silently ignores the problem and does not override the existing file, but assumes everything is okay. The Move option requires Camel 2. Existing option to be configured as well. The option eager.

Delete. Target. File can be used to control what to do if an moving the file, and there exists already an existing file, otherwise causing the move operation to fail. The Move option will move any existing files, before writing the target file.

Try. Rename. Camel 2. File. Name option is in use. This allows to try renaming the file from the temporary name to the actual name, without doing any exists check. This check may be faster on some file systems and especially FTP servers. Prefixnull. This option is used to write the file using a temporary name and then, after the write is complete, rename it to the real name.

Can be used to identify files being written and also avoid consumers (not using exclusive read locks) reading in progress files. Is often used by FTP when uploading big files. File. Namenull. Camel 2. The same as temp. Prefix option but offering a more fine grained control on the naming of the temporary filename as it uses the File Language.

Existingnull. Camel 2. Expression (such as File Language) used to compute file name to use when file. Exist=Move is configured. To move files into a backup subdirectory just enter backup. This option only supports the following File Language tokens: "file: name", "file: name.

Notice the "file: parent" is not supported by the FTP component, as the FTP component can only move any existing files to a relative directory based on current dir as base. Last. Modifiedfalse. Camel 2. 2: Will keep the last modified timestamp from the source file (if any).

Will use the Exchange. FILE_LAST_MODIFIED header to located the timestamp. This header can contain either a java. Date or long with the timestamp. If the timestamp exists and the option is enabled it will set this timestamp on the written file. Note: This option only applies to the file producer.

You cannot use this option with any of the ftp producers. Delete. Target. Filetrue.

Camel 2. 3: Whether or not to eagerly delete any existing target file. This option only applies when you use file. Exists=Override and the temp. File. Name option as well.

You can use this to disable (set it to false) deleting the target file before the temp file is written. For example you may write big files and want the target file to exists during the temp file is being written. This ensure the target file is only deleted until the very last moment, just before the temp file is being renamed to the target filename.

From Camel 2. 1. 0. Exist=Move is enabled, and an existing file exists. If this option copy. And. Delete. On. Rename. Failis false, then an exception will be thrown if an existing file existed, if its true, then the existing file is deleted before the move operation. File. Namenull. Camel 2. If provided, then Camel will write a 2nd done file when the original file has been written.

The done file will be empty. This option configures what file name to use.

Either you can specify a fixed name. Or you can use dynamic placeholders. The done file will always be written in the same folder as the original file.

See writing done file section for examples. Null. Bodyfalse. Camel 2. Used to specify if a null body is allowed during file writing. If set to true then an empty file will be created, when set to false, and attempting to send a null body to the file component, a Generic. File. Write. Exception of 'Cannot write null body to file.' will be thrown. If the `file. Exist` option is set to 'Override', then the file will be truncated, and if set to `append` the file will remain unchanged. Writestrue. Camel 2.

Whether to force syncing writes to the file system. You can turn this off if you do not want this level of guarantee, for example if writing to logs / audit logs etc; this would yield better performance. Camel 2. 1. 5. 0: Specify the file permissions which is sent by the producer, the chmod value must be between 0. If there is a leading digit like in 0.

Default behavior for file producer. By default it will override any existing file, if one exist with the same name. Move and Delete operations. Any move or delete operations is executed after (post command) the routing has completed; so during processing of the Exchange the file is still located in the inbox folder.

Lets illustrate this with an example: When a file is dropped in the inbox folder, the file consumer notices this and creates a new File. Exchange that is routed to the handle. Order bean. The bean then processes the File object. At this point in time the file is still located in the inbox folder. After the bean completes, and thus the route is completed, the file consumer will perform the move operation and move the file to the .

The move and pre. Move options is considered as a directory name (though if you use an expression such as File Language, or Simple then the result of the expression evaluation is the file name to be used - eg if you setthen that's using the File Language which we use return the file name to be used), which can be either relative or absolute.

If relative, the directory is created as a sub- folder from within the folder where the file was consumed. By default, Camel will move consumed files to the . If you want to delete the file after processing, the route should be: We have introduced a pre move operation to move files before they are processed. This allows you to mark which files have been scanned as they are moved to this sub folder before being processed. You can combine the pre move and the regular move: So in this situation, the file is in the inprogress folder when being processed and after it's processed, it's moved to the . Fine grained control over Move and Pre. Move option. The move and pre.

Move option is Expression- based, so we have the full power of the File Language to do advanced configuration of the directory and name pattern. Camel will, in fact, internally convert the directory name you enter into a File Language expression. So when we enter move=.

Camel will convert this into: ${file: parent}/. This is only done if Camel detects that you have not provided a ${ } in the option value yourself. So when you enter a ${ } Camel will not convert it and thus you have the full power. So if we want to move the file into a backup folder with today's date as the pattern, we can do: About move. Failed. The move. Failed option allows you to move files that could not be processed succesfully to another location such as a error folder of your choice. For example to move the files in an error folder with a timestamp you can use move.

Failed=/error/${file: name. MMdd. HHmmss. SSS}.${file: ext}. See more examples at File Language. The following headers are supported by this component: File producer only. Header. Description. Camel. File. Name.

Specifies the name of the file to write (relative to the endpoint directory). The name can be a String; a String with a File Language or Simple expression; or an Expression object. If it's null then Camel will auto- generate a filename based on the message unique ID.

Camel. File. Name. Produced. The actual absolute filepath (path + name) for the output file that was written. This header is set by Camel and its purpose is providing end- users with the name of the file that was written. Camel. Overrule. File. Name. Camel 2. 1.

Is used for overruling Camel. File. Name header and use the value instead (but only once, as the producer will remove this header after writing the file). The value can be only be a String. Notice that if the option file. Name has been configured, then this is still being evaluated. File consumer only. Header. Description.

Camel. File. Name. Name of the consumed file as a relative file path with offset from the starting directory configured on the endpoint. Camel. File. Name. Only. Only the file name (the name with no leading paths).

Camel. File. Absolute. A boolean option specifying whether the consumed file denotes an absolute path or not. Should normally be false for relative paths. Absolute paths should normally not be used but we added to the move option to allow moving files to absolute paths.

But can be used elsewhere as well. Camel. File. Absolute. Path. The absolute path to the file.

For relative files this path holds the relative path instead. Camel. File. Path. The file path. For relative files this is the starting directory + the relative filename.

For absolute files this is the absolute path. Camel. File. Relative. Path. The relative path. Camel. File. Parent. The parent path. Camel. File. Length. A long value containing the file size. Camel. File. Last.

Modified. A Long value containing the last modified timestamp of the file. In Camel 2. 1. 0. Date. Batch Consumer. This component implements the Batch Consumer. Exchange Properties, file consumer only. As the file consumer is Batch. Consumer it supports batching the files it polls.

By batching it means that Camel will add some properties to the Exchange so you know the number of files polled the current index in that order. Property. Description. Camel. Batch. Size. The total number of files that was polled in this batch.

Camel. Batch. Index. The current index of the batch. Starts from 0. Camel. Batch. Complete. A boolean value indicating the last Exchange in the batch. Is only true for the last entry. This allows you for instance to know how many files exists in this batch and for instance let the Aggregator. Using charset. Available as of Camel 2.

The charset option allows for configuring an encoding of the files on both the consumer and producer endpoints. For example if you read utf- 8 files, and want to convert the files to iso- 8. You can also use the convert.

Body. To in the route. In the example below we have still input files in utf- 8 format, but we want to convert the file content to a byte array in iso- 8. And then let a bean process the data. Before writing the content to the outbox folder using the current charset. If you omit the charset on the consumer endpoint, then Camel does not know the charset of the file, and would by default use "UTF- 8".

However you can configure a JVM system property to override and use a different default encoding with the key org. In the example below this could be a problem if the files is not in UTF- 8 encoding, which would be the default encoding for read the files. In this example when writing the files, the content has already been converted to a byte array, and thus would write the content directly as is (without any further encodings). You can also override and control the encoding dynamic when writing files, by setting a property on the exchange with the key Exchange. CHARSET_NAME. For example in the route below we set the property with a value from a message header. We suggest to keep things simpler, so if you pickup files with the same encoding, and want to write the files in a specific encoding, then favor to use the charset option on the endpoints.

Notice that if you have explicit configured a charset option on the endpoint, then that configuration is used, regardless of the Exchange. CHARSET_NAME property.

If you have some issues then you can enable DEBUG logging on org. Camel logs when it reads/write a file using a specific charset. For example the route below will log the following: And the logs: Common gotchas with folder and filenames. When Camel is producing files (writing files) there are a few gotchas affecting how to set a filename of your choice. By default, Camel will use the message ID as the filename, and since the message ID is normally a unique generated ID, you will end up with filenames such as: ID- MACHINENAME- 2. If such a filename is not desired, then you must provide a filename in the Camel. File. Name message header.

The constant, Exchange. FILE_NAME, can also be used. The sample code below produces files using the message ID as the filename: To use report. Camel. File. Name: And a syntax where we set the filename on the endpoint with the file.

Name URI option. Filename Expression. Filename can be set either using the expression option or as a string- based File Language expression in the Camel.

File. Name header. See the File Language for syntax and samples. Consuming files from folders where others drop files directly. Beware if you consume files from a folder where other applications write files directly. Take a look at the different read. Lock options to see what suits your use cases.

The best approach is however to write to another folder and after the write move the file in the drop folder. However if you write files directly to the drop folder then the option changed could better detect whether a file is currently being written/copied as it uses a file changed algorithm to see whether the file size / modification changes over a period of time. The other read lock options rely on Java File API that sadly is not always very good at detecting this. You may also want to look at the done. File. Name option, which uses a marker file (done) to signal when a file is done and ready to be consumed.

Using done files. Available as of Camel 2. See also section writing done files below. If you want only to consume files when a done file exists, then you can use the done. File. Name option on the endpoint. Will only consume files from the bar folder, if a file name done exists in the same directory as the target files. Camel will automatically delete the done file when it's done consuming the files.

From Camel 2. 9. 3 onwards Camel will not automatic delete the done file if noop=true is configured. However its more common to have one done file per target file. This means there is a 1: 1 correlation. To do this you must use dynamic placeholders in the done. File. Name option. Currently Camel supports the following two dynamic tokens: file: name and file: name. The consumer only supports the static part of the done file name as either prefix or suffix (not both).

In this example only files will be polled if there exists a done file with the name file name. For examplehello.

You can also use a prefix for the done file, such as: hello. Writing done files.

Available as of Camel 2. After you have written af file you may want to write an additional done file as a kinda of marker, to indicate to others that the file is finished and has been written. To do that you can use the done. File. Name option on the file producer endpoint. Will simply create a file named done in the same directory as the target file.

However its more common to have one done file per target file. This means there is a 1: 1 correlation. To do this you must use dynamic placeholders in the done.

File. Name option. Currently Camel supports the following two dynamic tokens: file: name and file: name.

Will for example create a file named done- foo. Will for example create a file named foo. Will for example create a file named foo. Samples. Read from a directory and write to another directory.

Read from a directory and write to another directory using a overrule dynamic name. Listen on a directory and create a message for each file dropped there. Copy the contents to the outputdir and delete the file in the inputdir.

Reading recursively from a directory and writing to another. Listen on a directory and create a message for each file dropped there. Copy the contents to the outputdir and delete the file in the inputdir. Will scan recursively into sub- directories. Will lay out the files in the same directory structure in the outputdir as the inputdir, including any sub- directories. Will result in the following output layout: Using flatten.

If you want to store the files in the outputdir directory in the same directory, disregarding the source directory layout (e. Will result in the following output layout: Reading from a directory and the default move operation. Camel will by default move any processed file into a . Affects the layout as follows: beforeafter.

Read from a directory and process the message in java. The body will be a File object that points to the file that was just dropped into the inputdir directory. Writing to files.

Camel is of course also able to write files, i. In the sample below we receive some reports on the SEDA queue that we process before they are written to a directory. Write to subdirectory using Exchange.

FILE_NAMEUsing a single route, it is possible to write a file to any number of subdirectories. If you have a route setup as such: You can have my.

Bean set the header Exchange. FILE_NAME to values such as: This allows you to have a single route to write files to multiple destinations. Writing file through the temporary directory relative to the final destination. Sometime you need to temporarily write the files to some directory relative to the destination directory. Such situation usually happens when some external process with limited filtering capabilities is reading from the directory you are writing to. In the example below files will be written to the  /var/myapp/files. In. Progress directory and after data transfer is done, they will be atomically moved to the /var/myapp/final.

Directory directory. Using expression for filenames. In this sample we want to move consumed files to a backup folder using today's date as a sub- folder name: See File Language for more samples. Avoiding reading the same file more than once (idempotent consumer)Camel supports Idempotent Consumer directly within the component so it will skip already processed files. This feature can be enabled by setting the idempotent=true option. Camel uses the absolute file name as the idempotent key, to detect duplicate files. From Camel 2. 1. 1 onwards you can customize this key by using an expression in the idempotent.

Key option. For example to use both the name and the file size as the key. By default Camel uses a in memory based store for keeping track of consumed files, it uses a least recently used cache holding up to 1. You can plugin your own implementation of this store by using the idempotent. Repository option using the # sign in the value to indicate it's a referring to a bean in the Registry with the specified id. Camel will log at DEBUG level if it skips a file because it has been consumed before: Using a file based idempotent repository.

In this section we will use the file based idempotent repository org. File. Idempotent. Repository instead of the in- memory based that is used as default.

This repository uses a 1st level cache to avoid reading the file repository. It will only use the file repository to store the content of the 1st level cache. Thereby the repository can survive server restarts. It will load the content of the file into the 1st level cache upon startup. The file structure is very simple as it stores the key in separate lines in the file. By default, the file store has a size limit of 1mb.

When the file grows larger Camel will truncate the file store, rebuilding the content by flushing the 1st level cache into a fresh empty file. We configure our repository using Spring XML creating our file idempotent repository and define our file consumer to use our repository with the idempotent. Repository using # sign to indicate Registry lookup: Using a JPA based idempotent repository. In this section we will use the JPA based idempotent repository instead of the in- memory based that is used as default.

First we need a persistence- unit in META- INF/persistence. Message. Processed as model. Then we need to setup a Spring jpa. Template in the spring XML file: Error formatting macro: snippet: java. Index. Out. Of. Bounds. Exception: Index: 2.

Size: 2. 0And finally we can create our JPA idempotent repository in the spring XML file as well: And yes then we just need to refer to the jpa. Store bean in the file consumer endpoint using the idempotent. Repository using the # syntax option: Filter using org. Generic. File. Filter. Camel supports pluggable filtering strategies.

You can then configure the endpoint with such a filter to skip certain files being processed. In the sample we have built our own filter that skips files starting with skip in the filename: And then we can configure our route using the filter attribute to reference our filter (using # notation) that we have defined in the spring XML file: Filtering using ANT path matcher. New options from Camel 2. There are now ant.

Include and ant. Exclude options to make it easy to specify ANT style include/exclude without having to define the filter. See the URI options above for more information. The ANT path matcher is shipped out- of- the- box in the camel- spring jar. So you need to depend on camel- spring if you are using Maven.

The reasons is that we leverage Spring's Ant. Path. Matcher to do the actual matching. The file paths is matched with the following rules: ?

The sample below demonstrates how to use it: Sorting using Comparator. Camel supports pluggable sorting strategies.

This strategy it to use the build in java. Comparator in Java. You can then configure the endpoint with such a comparator and have Camel sort the files before being processed. In the sample we have built our own comparator that just sorts by file name: And then we can configure our route using the sorter option to reference to our sorter (my.

Sorter) we have defined in the spring XML file: URI options can reference beans using the # syntax. In the Spring DSL route about notice that we can refer to beans in the Registry by prefixing the id with #. So writing sorter=#my. Sorter, will instruct Camel to go look in the Registry for a bean with the ID, my. Sorter. Sorting using sort. By. Camel supports pluggable sorting strategies.

This strategy it to use the File Language to configure the sorting. The sort. By option is configured as follows: Where each group is separated with semi colon. In the simple situations you just use one group, so a simple example could be: This will sort by file name, you can reverse the order by prefixing reverse: to the group, so the sorting is now Z.

A: As we have the full power of File Language we can use some of the other parameters, so if we want to sort by file size we do: You can configure to ignore the case, using ignore. Case: for string comparison, so if you want to use file name sorting but to ignore the case then we do: You can combine ignore case and reverse, however reverse must be specified first: In the sample below we want to sort by last modified file, so we do: And then we want to group by name as a 2nd option so files with same modifcation is sorted by name: Now there is an issue here, can you spot it? Well the modified timestamp of the file is too fine as it will be in milliseconds, but what if we want to sort by date only and then subgroup by name? Well as we have the true power of File Language we can use the its date command that supports patterns. So this can be solved as: Yeah, that is pretty powerful, oh by the way you can also use reverse per group, so we could reverse the file names: Using Generic. File. Process. Strategy.

The option process. Strategy can be used to use a custom Generic. File. Process. Strategy that allows you to implement your own begin, commit and rollback logic. For instance lets assume a system writes a file in a folder you should consume. But you should not start consuming the file before another ready file has been written as well.

So by implementing our own Generic. File. Process. Strategy we can implement this as: In the begin() method we can test whether the special ready file exists. The begin method returns a boolean to indicate if we can consume the file or not. In the abort() method (Camel 2. Using filter. The filter option allows you to implement a custom filter in Java code by implementing the org. Generic. File. Filter interface. This interface has an accept method that returns a boolean.

Return true to include the file, and false to skip the file. From Camel 2. 1. 0 onwards, there is a is. Directory method on Generic. File whether the file is a directory. This allows you to filter unwanted directories, to avoid traversing down unwanted directories.

For example to skip any directories which starts with "skip" in the name, can be implemented as follows: The file and ftp consumers, will by default try to pickup files. Only if that is successful then a Camel Exchange can be created and passed in the Camel routing engine. When the Exchange is processed by the routing engine, then the Camel Error Handling takes over (eg the on.

Exception / error. Handler in the routes). However outside the scope of the routing engine, any exceptions handling is component specific. Camel offers a org. Exception. Handler that allows components to use that as a pluggable hook for end users to use their own implementation. Camel offers a default Logging.

Exception. Handler that will log the exception at ERROR/WARN level. For the file and ftp components this would be the case. However if you want to bridge the Exception.

Handler so it uses the Camel Error Handling, then you need to implement a custom Exception. Handler that will handle the exception by creating a Camel Exchange and send it to the routing engine; then the error handling of the routing engine can get triggered. Easier with Camel 2. The new option consumer.

Error. Handler can be set to true, to make this even easier. See further below for more details. Here is such an example based upon an unit test. First we have a custom Exception. Handler where you can see we deal with the exception by sending it to a Camel Endpoint named "direct: file- error": Then we have a Camel route that uses the Camel routing error handler, which is the on. Exception where we handle any IOException being thrown. We then send the message to the same "direct: file- error" endpoint, where we handle it by transforming it to a message, and then being sent to a Mock endpoint.

This is just for testing purpose. You can handle the exception in any custom way you want, such as using a Bean or sending an email etc. Notice how we configure our custom My. Exception. Handler by using the consumer. Handler option to refer to #my. Exception. Handler which is a id of the bean registered in the Registry.

If using Spring XML or OSGi Blueprint, then that would be a < bean id="my. Exception. Handler" class="com. My. Exception. Handler"/> :The source code for this example can be seen here.

Using consumer. bridge. Error. Handler. Available as of Camel 2. If you want to use the Camel Error Handler to deal with any exception occurring in the file consumer, then you can enable the consumer.

Error. Handler option as shown below: So all you have to do is to enable this option, and the error handler in the route will take it from there. Important when using consumer. Error. Handler. When using consumer. Error. Handler, then interceptors, On. Completions does not apply. The Exchange is processed directly by the Camel Error Handler, and does not allow prior actions such as interceptors, on. Completion to take action.

Debug logging. This component has log level TRACE that can be helpful if you have problems.