# changelog

author: Tim Theisen <tim@cs.wisc.edu> - 10.0.6-1
changelog:
  - In SSL Authentication, use the identity instead of the X.509 proxy subject
  - Can use environment variable to locate the client's SSL X.509 credential
  - ClassAd aggregate functions now tolerate undefined values
  - Fix Python binding bug where accounting ads were omitted from the result
  - The Python bindings now properly report the HTCondor version
  - remote_initial_dir works when submitting a grid batch job remotely via ssh
  - Add a ClassAd stringlist subset match function


author: Tim Theisen <tim@cs.wisc.edu> - 10.6.0-1
changelog:
  - Administrators can enable and disable job submission for a specific user
  - Work around memory leak in libcurl on EL7 when using the ARC-CE GAHP
  - Container images may now be transferred via a file transfer plugin
  - Add ClassAd stringlist subset match function
  - Add submit file macro '$(JobId)' which expands to full ID of the job
  - The job's executable is no longer renamed to 'condor_exec.exe'


author: Tim Theisen <tim@cs.wisc.edu> - 10.0.7-1
changelog:
  - Fixed bug where held condor cron jobs would never run when released
  - Improved daemon IDTOKENS logging to make useful messages more prominent
  - Remove limit on certificate chain length in SSL authentication
  - condor_config_val -summary now works with a remote configuration query
  - Prints detailed message when condor_remote_cluster fails to fetch a URL
  - Improvements to condor_preen


author: Tim Theisen <tim@cs.wisc.edu> - 10.7.0-1
changelog:
  - Support for Debian 12 (Bookworm)
  - Can run defrag daemons with different policies on distinct sets of nodes
  - Added want_io_proxy submit command
  - Apptainer is now included in the HTCondor tarballs
  - Fix 10.5.0 bug where reported CPU time is very low when using cgroups v1
  - Fix 10.5.0 bug where .job.ad and .machine.ad were missing for local jobs


author: Tim Theisen <tim@cs.wisc.edu> - 10.0.8-1
changelog:
  - Avoid kernel panic on some Enterprise Linux 8 systems
  - Fix bug where early termination of service nodes could crash DAGMan
  - Limit email about long file transfer queue to once daily
  - Various fixes to condor_adstash


author: Tim Theisen <tim@cs.wisc.edu> - 10.8.0-1
changelog:
  - Fold the classads, blahp, and procd RPMs into the main condor RPM
  - Align the Debian packages and package names with the RPM packaging
  - On Linux, the default configuration enforces memory limits with cgroups
  - condor_status -gpus shows nodes with GPUs and the GPU properties
  - condor_status -compact shows a row for each slot type
  - New ENV command controls which environment variables are present in DAGMan


author: Tim Theisen <tim@cs.wisc.edu> - 10.0.9-1
changelog:
  - The condor_upgrade_check script now provides guidance on updating to 23.0
  - The htchirp Python binding now properly locates the chirp configuration
  - Fix bug that prevented deletion of HTCondor passwords on Windows


author: Tim Theisen <tim@cs.wisc.edu> - 10.0.3-1
changelog:
  - GPU metrics continues to be reported after the startd is reconfigured
  - Fixed issue where GPU metrics could be wildly over-reported
  - Fixed issue that kept jobs from running when installed on Debian or Ubuntu
  - Fixed DAGMan problem when retrying a proc failure in a multi-proc node


author: Tim Theisen <tim@cs.wisc.edu> - 10.4.0-1
changelog:
  - DAGMan no longer carries the entire environment into the DAGMan job
  - Allows EGI CheckIn tokens to be used the with SciTokens authentication


author: Tim Theisen <tim@cs.wisc.edu> - 10.4.2-1
changelog:
  - Fix bug where remote submission of batch grid universe jobs fail
  - Fix bug where HTCondor-CE fails to handle jobs after HTCondor restarts


author: Tim Theisen <tim@cs.wisc.edu> - 10.0.4-1
changelog:
  - Provides script to assist updating from HTCondor version 9 to version 10
  - Fixes a bug where rarely an output file would not be transferred back
  - Fixes counting of submitted jobs, so MAX_JOBS_SUBMITTED works correctly
  - Fixes SSL Authentication failure when PRIVATE_NETWORK_NAME was set
  - Fixes rare crash when SSL or SCITOKENS authentication was attempted
  - Can allow client to present an X.509 proxy during SSL authentication
  - Fixes issue where a users jobs were ignored by the HTCondor-CE on restart
  - Fixes issues where some events that HTCondor-CE depends on were missing


author: Tim Theisen <tim@cs.wisc.edu> - 10.5.0-1
changelog:
  - Can now define DAGMan save points to be able to rerun DAGs from there
  - Expand default list of environment variables passed to the DAGMan manager
  - Administrators can prevent users using "getenv = true" in submit files
  - Improved throughput when submitting a large number of ARC-CE jobs
  - Execute events contain the slot name, sandbox path, resource quantities
  - Can add attributes of the execution point to be recorded in the user log
  - Enhanced condor_transform_ads tool to ease offline job transform testing
  - Fixed a bug where memory limits over 2 GiB might not be correctly enforced


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.11-1
changelog:
  - The Job Router can now create an IDTOKEN for use by the job
  - Fix bug where a self-checkpointing job may erroneously be held
  - Fix bug where the Job Router could erroneously substitute a default value
  - Fix bug where a file transfer error may identify the wrong file
  - Fix bug where condor_ssh_to_job may fail to connect


author: Tim Theisen <tim@cs.wisc.edu> - 9.7.0-1
changelog:
  - Support environment variables, other application elements in ARC REST jobs
  - Container universe supports Singularity jobs with hard-coded command
  - DAGMan submits jobs directly (does not shell out to condor_submit)
  - Meaningful error message and sub-code for file transfer failures
  - Add file transfer statistics for file transfer plugins
  - Add named list policy knobs for SYSTEM_PERIODIC_ policies


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.12-1
changelog:
  - Fix bug in parallel universe that could cause the schedd to crash
  - Fix rare crash where a daemon tries to use a discarded security session


author: Tim Theisen <tim@cs.wisc.edu> - 9.8.0-1
changelog:
  - Support for Heterogeneous GPUs, some configuration required
  - Allow HTCondor to utilize grid sites requiring two-factor authentication
  - Technology preview: bring your own resources from (some) NSF HPC clusters


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.13-1
changelog:
  - Schedd and startd cron jobs can now log output upon non-zero exit
  - condor_config_val now produces correct syntax for multi-line values
  - The condor_run tool now reports submit errors and warnings to the terminal
  - Fix issue where Kerberos authentication would fail within DAGMan
  - Fix HTCondor startup failure with certain complex network configurations


author: Tim Theisen <tim@cs.wisc.edu> - 9.9.0-1
changelog:
  - A new authentication method for remote HTCondor administration
  - Several changes to improve the security of connections
  - Fix issue where DAGMan direct submission failed when using Kerberos
  - The submission method is now recorded in the job ClassAd
  - Singularity jobs can now pull from Docker style repositories
  - The OWNER authorization level has been folded into the ADMINISTRATOR level


author: Mátyás Selmeci <matyas@cs.wisc.edu> - 8.9.11-1.4
changelog:
  - Add SOFTWARE-4525-condor_watch_q_crash.patch to fix a condor_watch_q crash when watching DAGs (SOFTWARE-4525)
  - Add HTCONDOR-315-Always-handle-exceptions-when-decoding-.patch to fix a schedd crash caused by malformed SciTokens (SOFTWARE-4533)


author: Tim Theisen <tim@cs.wisc.edu> - 8.8.13-1
changelog:
  - condor_ssh_to_job now maps CR and NL to work with editors like nano
  - Improved the performance of data transfer in condor_ssh_to_job
  - HA replication now accepts SHA-2 checksums to prepare for MD5 removal
  - Submission to NorduGrid ARC CE works with newer ARC CE versions
  - Fixed condor_annex crashes on platforms with newer compilers
  - Fixed "use feature: GPUsMonitor" to locate the monitor binary on Windows
  - Fixed a bug that prevented using the '@' character in an event log path


author: Tim Theisen <tim@cs.wisc.edu> - 8.9.13-1
changelog:
  - Host based security is no longer the default security model
  - Hardware accelerated integrity and AES encryption used by default
  - Normally, AES encryption is used for all communication and file transfers
  - Fallback to Triple-DES or Blowfish when interoperating with older versions
  - Simplified and automated new HTCondor installations
  - HTCondor now detects instances of multi-instance GPUs
  - Fixed memory leaks (collector updates in 8.9 could leak a few MB per day)
  - Many other enhancements and bug fixes, see version history for details


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.0-1
changelog:
  - Absent any configuration, HTCondor denies authorization to all users
  - AES encryption is used for all communication and file transfers by default
  - New IDTOKEN authentication method enables fine-grained authorization
  - IDTOKEN authentication method is designed to replace GSI
  - Improved support for GPUs, including machines with multiple GPUs
  - New condor_watch_q tool that efficiently provides live job status updates
  - Many improvements to the Python bindings
  - New Python bindings for DAGMan and chirp
  - Improved file transfer plugins supporting uploads and authentication
  - File transfer times are now recorded in the job log
  - Added support for jobs that need to acquire and use OAUTH tokens
  - Many memory footprint and performance improvements in DAGMan
  - Submitter ceilings can limit the number of jobs per user in a pool


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.0-1.3
changelog:
  - Fix malformed default Bosco tarball URL format (HTCONDOR-433)
  - Fix autocluster problem (HTCONDOR-414)
  - Fix claim activation faiure (HTCONDOR-434)
  - Fix Bosco tarball extraction (HTCONDOR-438)


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.5-1
changelog:
  - Other authentication methods are tried if mapping fails using SciTokens
  - Fix rare crashes from successful condor_submit, which caused DAGMan issues
  - Fix bug where ExitCode attribute would be suppressed when OnExitHold fired
  - condor_who now suppresses spurious warnings coming from netstat
  - The online manual now has detailed instructions for installing on MacOS
  - Fix bug where misconfigured MIG devices confused condor_gpu_discovery
  - The transfer_checkpoint_file list may now include input files


author: Tim Theisen <tim@cs.wisc.edu> - 9.1.3-1
changelog:
  - Globus GSI is no longer needed for X.509 proxy delegation
  - Globus GSI authentication is disabled by default
  - The job ad now contains a history of job holds and hold reasons
  - If a user job policy expression evaluates to undefined, it is ignored


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.6-1
changelog:
  - CUDA_VISIBLE_DEVICES can now contain GPU-<uuid> formatted values
  - Fixed a bug that caused jobs to fail when using newer Singularity versions
  - Fixed a bug in the Windows MSI installer for the latest Windows 10 version
  - Fixed bugs relating to the transfer of standard out and error logs
  - MacOS 11.x now reports as 10.16.x (which is better than reporting x.0)


author: Tim Theisen <tim@cs.wisc.edu> - 9.2.0-1
changelog:
  - Add SERVICE node that runs alongside the DAG for the duration of the DAG
  - Fix problem where proxy delegation to older HTCondor versions failed
  - Jobs are now re-run if the execute directory unexpectedly disappears
  - HTCondor counts the number of files transfered at the submit node
  - Fix a bug that caused jobs to fail when using newer Singularity versions


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.7-1
changelog:
  - Fix bug where condor_gpu_discovery could crash with older CUDA libraries
  - Fix bug where condor_watch_q would fail on machines with older kernels
  - condor_watch_q no longer has a limit on the number of job event log files
  - Fix bug where a startd could crash claiming a slot with p-slot preemption
  - Fix bug where a job start would not be recorded when a shadow reconnects


author: Tim Theisen <tim@cs.wisc.edu> - 9.3.0-1
changelog:
  - Discontinue support for Globus GSI
  - Discontinue support for grid type 'nordugrid', use 'arc' instead
  - MacOS version strings now include the major version number (10 or 11)
  - File transfer plugin sample code to aid in developing new plugins
  - Add generic knob to set the slot user for all slots


author: Tim Theisen <tim@cs.wisc.edu> - 9.3.2-1
changelog:
  - Add allowed_execute_duration condor_submit command to cap job run time
  - Fix bug where self check-pointing jobs may be erroneously held


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.8-1
changelog:
  - Fix bug where huge values of ImageSize and others would end up negative
  - Fix bug in how MAX_JOBS_PER_OWNER applied to late materialization jobs
  - Fix bug where the schedd could choose a slot with insufficient disk space
  - Fix crash in ClassAd substr() function when the offset is out of range
  - Fix bug in Kerberos code that can crash on macOS and could leak memory
  - Fix bug where a job is ignored for 20 minutes if the startd claim fails


author: Tim Theisen <tim@cs.wisc.edu> - 9.4.0-1
changelog:
  - Initial implementation of Job Sets in the htcondor CLI tool
  - The access point administrator can add keywords to the submit language
  - Add submit commands that limit job run time
  - Fix bug where self check-pointing jobs may be erroneously held


author: Tim Theisen <tim@cs.wisc.edu> - 8.8.16-1
changelog:
  - Fix for security issue
  - https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2022-0003


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.10-1
changelog:
  - Fixes for security issues
  - https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2022-0001
  - https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2022-0002
  - https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2022-0003


author: Tim Theisen <tim@cs.wisc.edu> - 9.1.1-1
changelog:
  - Fixes for security issues
  - https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2021-0003.html
  - https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2021-0004.html


author: Tim Theisen <tim@cs.wisc.edu> - 8.8.15-1
changelog:
  - Fix for security issue
  - https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2021-0003/


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.4-1
changelog:
  - Fixes for security issues
  - https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2021-0003/
  - https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2021-0004/


author: Tim Theisen <tim@cs.wisc.edu> - 10.2.1-1
changelog:
  - Improved condor_schedd scalability when a user runs more than 1,000 jobs
  - Fix issue where condor_ssh_to_job fails if the user is not in /etc/passwd
  - The Python Schedd.query() now returns the ServerTime attribute
  - Fixed issue that prevented HTCondor installation on Ubuntu 18.04


author: John Knoeller <johnkn@cs.wisc.edu> - 10.0.2-1
changelog:
  - HTCondor can optionally create intermediate directories for output files
  - Improved condor_schedd scalability when a user runs more than 1,000 jobs
  - Fix issue where condor_ssh_to_job fails if the user is not in /etc/passwd
  - The Python Schedd.query() now returns the ServerTime attribute for Fifemon
  - VM Universe jobs pass through the host CPU model to support newer kernels
  - HTCondor Python wheel is now available for Python 3.11
  - Fix issue that prevented HTCondor installation on Ubuntu 18.04


author: Tim Theisen <tim@cs.wisc.edu> - 10.3.0-1
changelog:
  - Now evicts OOM killed jobs when they are under their requested memory
  - HTCondor glideins can now use cgroups if one has been prepared
  - Can write job information in an AP history file for each execution attempt
  - Can now specify a lifetime for condor_gangliad metrics
  - The condor_schedd now advertises a count of unmaterialized jobs


author: Tim Theisen <tim@cs.wisc.edu> - 10.0.5-1
changelog:
  - Rename upgrade9to10checks.py script to condor_upgrade_check
  - Fix spurious warning from condor_upgrade_check about regexes with spaces


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.18-1
changelog:
  - Can configure clients to present an X.509 proxy during SSL authentication
  - Provides script to assist updating from HTCondor version 9 to version 10


author: Tim Theisen <tim@cs.wisc.edu> - 9.4.1-1
changelog:
  - Add the ability to track slot activation metrics
  - Fix bug where a file transfer plugin failure code may not be reported


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.9-1
changelog:
  - Added Debian 11 (bullseye) as a supported platform
  - Since CentOS 8 has reached end of life, we build and test on Rocky Linux 8
  - The OAUTH credmon is now packaged for Enterprise Linux 8


author: Tim Theisen <tim@cs.wisc.edu> - 9.5.0-1
changelog:
  - Initial implementation of Container Universe
  - HTCondor will automatically detect container type and where it can run
  - The blahp is no longer separate, it is now an integral part of HTCondor
  - Docker Universe jobs can now self-checkpoint
  - Added Debian 11 (bullseye) as a supported platform
  - Since CentOS 8 has reached end of life, we build and test on Rocky Linux 8


author: Tim Theisen <tim@cs.wisc.edu> - 9.5.2-1
changelog:
  - Fix bug where job may not go on hold when exceeding allowed_job_duration
  - Fix bug where the condor_shadow could run indefinitely
  - Fix bug where condor_ssh_to_job may fail to connect
  - Fix bug where a file transfer error may identify the wrong file


author: Tim Theisen <tim@cs.wisc.edu> - 9.5.3-1
changelog:
  - Fix daemon crash where one of multiple collectors is not in DNS
  - Fix bug where initial schedd registration was rarely delayed by an hour
  - Can set CCB_TIMEOUT and choose to not start up if CCB address unavailable


author: Tim Theisen <tim@cs.wisc.edu> - 9.5.4-1
changelog:
  - The access point more robustly detects execution points that disappear
  - The condor_procd will now function if /proc is mounted with hidepid=2


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.14-1
changelog:
  - SciToken mapping failures are now recorded in the daemon logs
  - Fix bug that stopped file transfers when output and error are the same
  - Ensure that the Python bindings version matches the installed HTCondor
  - $(OPSYSANDVER) now expand properly in job transforms
  - Fix bug where context managed Python htcondor.SecMan sessions would crash
  - Fix bug where remote CPU times would rarely be set to zero


author: Tim Theisen <tim@cs.wisc.edu> - 9.10.0-1
changelog:
  - With collector administrator access, can manage all HTCondor pool daemons
  - SciTokens can now be used for authentication with ARC CE servers
  - Preliminary support for ARM and POWER RC on AlmaLinux 8
  - Prevent negative values when using huge files with a file transfer plugin


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.16-1
changelog:
  - Singularity now mounts /tmp and /var/tmp under the scratch directory
  - Fix bug where Singularity jobs go on hold at the first checkpoint
  - Fix bug where gridmanager deletes the X.509 proxy file instead of the copy
  - Fix file descriptor leak when using SciTokens for authentication


author: Tim Theisen <tim@cs.wisc.edu> - 9.11.0-1
changelog:
  - Modified GPU attributes to support the new 'require_gpus' submit command
  - Add (PREEMPT|HOLD)_IF_DISK_EXCEEDED configuration templates
  - ADVERTISE authorization levels now also provide READ authorization
  - Periodic release expressions no longer apply to manually held jobs
  - If a #! interpreter doesn't exist, a proper hold and log message appears
  - Can now set the Singularity target directory with 'container_target_dir'
  - If SciToken and X.509 available, uses SciToken for arc job authentication


author: Tim Theisen <tim@cs.wisc.edu> - 9.0.17-1
changelog:
  - Fix file descriptor leak when schedd fails to launch scheduler jobs
  - Fix failure to forward batch grid universe job's refreshed X.509 proxy
  - Fix DAGMan failure when the DONE keyword appeared in the JOB line
  - Fix HTCondor's handling of extremely large UIDs on Linux
  - Fix bug where OAUTH tokens lose their scope and audience upon refresh
  - Support for Apptainer in addition to Singularity


author: Tim Theisen <tim@cs.wisc.edu> - 9.12.0-1
changelog:
  - Provide a mechanism to bootstrap secure authentication within a pool
  - Add the ability to define submit templates
  - Administrators can now extend the help offered by condor_submit
  - Add DAGMan ClassAd attributes to record more information about jobs
  - On Linux, advertise the x86_64 micro-architecture in a slot attribute
  - Added -drain option to condor_off and condor_restart
  - Administrators can now set the shared memory size for Docker jobs
  - Multiple improvements to condor_adstash
  - HAD daemons now use SHA-256 checksums by default


author: Mátyás Selmeci <matyas@cs.wisc.edu> - 4.2.3-1.1
changelog:
  - Update to 4.2.3-1 from upstream (SOFTWARE-4885)
  - Add Check-for-DT_REG-macro.patch to avoid compile error on EL7 due to missing DT_REG macro


author: Mattias Ellert <mattias.ellert@physics.uu.se> - 5.0.1-1
changelog:
  - Update to version 5.0.1
  - Drop patch oidc-agent-webkit.patch (previously backported
  - Drop patch oidc-agent.patch (accepted upstram)


